Information processing device, information processing method, and program

ABSTRACT

To provide an information processing device, an information processing method, and a program capable of sharing a space while maintaining the degree of freedom of a visual line. An information processing device according to the present disclosure includes: a control unit configured to perform control in a manner that a display image generated based on image information which is generated through imaging of an imaging device mounted on a moving object moving in a space, imaging-device posture information which is information regarding a posture of the imaging device, and user view information which is obtained from a user manipulation device manipulated by a user and specifies a region that the user desires to view is displayed in a display region viewed by the user.

CROSS-REFERENCE TO PRIOR APPLICATION

This application is a continuation of U.S. patent application Ser. No.14/906,967 (filed on Jan. 22, 2016), which is a National Stage PatentApplication of PCT International Application No. PCT/JP2014/084350(filed on Dec. 25, 2014) under U.S.C. § 371, which claims priority toJapanese Patent Application Nos. 2014-191990 (filed on Sep. 19, 2014),2014-125799 (filed on Jun. 18, 2014), and 2014-028015 (filed on Feb. 17,2014), which are all hereby incorporated by reference in their entirety.

TECHNICAL FIELD

The present disclosure relates to an information processing device, aninformation processing method, and a program.

BACKGROUND ART

In recent years, to transfer human experiences to other people as theyare, first-person viewpoint images in wearable devices such ashead-mounted cameras are used, for example, to generate various kinds ofcontent. Interfaces for sharing experiences with other people or askingother people for knowledge or instructions by realizing communicationwith the other people through the foregoing transfer of the first-personviewpoint images have been proposed.

For example, Patent Literature 1 discloses a technology for transmittinga video imaged by an imaging device mounted on a head to another deviceso that the video can be watched with the other device.

CITATION LIST Patent Literature

Patent Literature 1: JP 2013-110764A

SUMMARY OF INVENTION Technical Problem

In the technology disclosed in Patent Literature 1, however, since avisual line of another person watching the transferred first-personviewpoint image is restricted to the visual line of a wearer wearing awearable device, the other person may not comprehend a space from adifferent viewpoint from the wearer.

Accordingly, it is desirable to provide an information processingdevice, an information processing method, and a program capable ofsharing a space while maintaining the degree of freedom of a visualline.

Solution to Problem

According to the present disclosure, there is provided an informationprocessing device including: a control unit configured to performcontrol in a manner that a display image generated based on imageinformation which is generated through imaging of an imaging devicemounted on a moving object moving in a space, imaging-device postureinformation which is information regarding a posture of the imagingdevice, and user view information which is obtained from a usermanipulation device manipulated by a user and specifies a region thatthe user desires to view is displayed in a display region viewed by theuser.

According to the present disclosure, there is provided an informationprocessing method including: performing control in a manner that adisplay image generated based on image information which is generatedthrough imaging of an imaging device mounted on a moving object movingin a space, imaging-device posture information which is informationregarding a posture of the imaging device, and user view informationwhich is obtained from a user manipulation device manipulated by a userand specifies a region that the user desires to view is displayed in adisplay region viewed by the user.

According to the present disclosure, there is provided a program causinga computer to realize a function of: performing control in a manner thata display image generated based on image information which is generatedthrough imaging of an imaging device mounted on a moving object movingin a space, imaging-device posture information which is informationregarding a posture of the imaging device, and user view informationwhich is obtained from a user manipulation device manipulated by a userand specifies a region that the user desires to view is displayed in adisplay region viewed by the user.

Advantageous Effects of Invention

According to the present disclosure described above, it is possible toshare a space while maintaining the degree of freedom of a visual line.

Note that the effects described above are not necessarily limited, andalong with or instead of the effects, any effect that is desired to beintroduced in the present specification or other effects that can beexpected from the present specification may be exhibited.

BRIEF DESCRIPTION OF DRAWINGS

FIG. 1 is an explanatory diagram illustrating a schematic configurationof a system according to a first embodiment of the present disclosure.

FIG. 2 is an explanatory diagram illustrating a schematic configurationof a device according to the embodiment.

FIG. 3 is an explanatory diagram schematically illustrating an exampleof a wearable device according to the embodiment.

FIG. 4A is a block diagram illustrating an example of the configurationof an information processing device according to the embodiment.

FIG. 4B is a block diagram illustrating an example of the configurationof an information processing device according to the embodiment.

FIG. 5 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 6 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 7 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 8 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 9 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 10 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 11 is a flowchart illustrating a flow example of an informationprocessing method according to the embodiment.

FIG. 12 is an explanatory diagram for describing a display controlprocess according to the embodiment.

FIG. 13 is an explanatory diagram for describing a display controlprocess according to the embodiment.

FIG. 14 is an explanatory diagram for describing a display controlprocess according to the embodiment.

FIG. 15 is an explanatory diagram for describing a display controlprocess according to the embodiment.

FIG. 16 is an explanatory diagram for describing a display controlprocess according to the embodiment.

FIG. 17 is a flowchart illustrating a flow example of a display controlprocess according to the embodiment.

FIG. 18 is a block diagram illustrating an example of the configurationof an information processing device according to a second embodiment ofthe present disclosure.

FIG. 19 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 20 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 21 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 22 is a flowchart illustrating a flow example of an informationprocessing method according to the embodiment.

FIG. 23 is an explanatory diagram for describing a function of theinformation processing device according to a modification example of theembodiment.

FIG. 24 is a flowchart illustrating a flow example of an informationprocessing method according to the modification example of theembodiment.

FIG. 25 is an explanatory diagram for describing a ray space.

FIG. 26 is an explanatory diagram illustrating a schematic configurationof a system according to a third embodiment of the present disclosure.

FIG. 27A is an explanatory diagram illustrating a schematicconfiguration of an imaging device according to the embodiment.

FIG. 27B is an explanatory diagram illustrating a schematicconfiguration of an imaging device according to the embodiment.

FIG. 28A is an explanatory diagram illustrating a schematicconfiguration of an imaging device according to the embodiment.

FIG. 28B is an explanatory diagram illustrating a schematicconfiguration of an imaging device according to the embodiment.

FIG. 29 is an explanatory diagram illustrating a schematic configurationof the imaging device according to the embodiment.

FIG. 30 is a block diagram illustrating an example of the configurationof an information processing device according to the embodiment.

FIG. 31 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 32A is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 32B is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 33 is an explanatory diagram for describing a function of theinformation processing device according to the embodiment.

FIG. 34 is a flowchart illustrating a flow example of an informationprocessing method according to the embodiment.

FIG. 35 is a flowchart illustrating a flow example of an informationprocessing method according to the embodiment.

DESCRIPTION OF EMBODIMENTS

Hereinafter, preferred embodiments of the present disclosure will bedescribed in detail with reference to the appended drawings. In thisspecification and the drawings, elements that have substantially thesame function and structure are denoted with the same reference signs,and repeated explanation is omitted.

The description will be made in the following order.

-   -   1. First embodiment    -   1.1 Example of system configuration    -   1.2 Configuration of information processing device    -   1.3 Flow of information processing method    -   1.4 Example of display control process    -   1.5 Conclusion    -   2. Second embodiment    -   2.1 Configuration of information processing device    -   2.2 Flow of information processing method    -   2.3 Modification examples of information processing method    -   2.4 Conclusion    -   3. Third embodiment    -   3.1 Example of system configuration    -   3.2 Configuration of imaging device    -   3.3 Configuration of information processing device    -   3.4 Flow of information processing method    -   3.5 Conclusion

First Embodiment

<Example of System Configuration>

FIG. 1 is an explanatory diagram illustrating a schematic configurationof a system according to a first embodiment of the present disclosure.As illustrated in FIG. 1, a system 10 according to the embodimentincludes a server 100 and clients 200 to 700.

The server 100 is a collective of functions realized by a single serverdevice or a plurality of server devices connected via various wired orwireless networks for cooperation. The server 100 provides variousservices to the client devices 200 to 700.

The client devices 200 to 700 are terminal devices connected with theserver 100 via various wired or wireless networks.

The server 100 and the client devices 200 to 700 independently orcooperatively realize the function of at least one of the following (1)to (7) in the system 10.

(1) A device that includes an imaging mechanism such as a camera andsupplies a captured image of the real space to the server 100 or theother client devices 200 to 700.

(2) A device that includes an imaging mechanism such as a camera,performs various kinds of image processing on a captured image of thereal space, and supplies various images related to the real space andobtained through the image processing to the server 100 or the otherclient devices 200 to 700.

(3) A device that includes an imaging mechanism such as a camera,performs various kinds of image processing on a captured image of thereal space, generates an image desired by a user according to variousmanipulations performed on various images by the user, and supplies thegenerated various images to the server 100 or the other client devices200 to 700.

(4) A device that includes at least a display mechanism such as adisplay, preferably further includes a manipulation mechanism such as atouch panel, acquires images supplied by the device (1), generatesimages desired by a user according to various manipulations performed onthe images by the user, and supplies the generated various images to theuser for the user to browse.

(5) A device that includes at least a display mechanism such as adisplay, preferably further includes a manipulation mechanism such as atouch panel, acquires images supplied by the device (2), generatesimages desired by a user according to various manipulations performed onthe images by the user, and supplies the generated various images to theuser for the user to browse.

(6) A device that includes at least a display mechanism such as adisplay, preferably further includes a manipulation mechanism such as atouch panel, acquires images supplied by the device (3) and supplies theimages to the user for the user to browse, and receives variousmanipulations on the images performed by the user.

(7) A device that includes a display mechanism such as a display anddisplays various images generated based on various user manipulationsreceived by the devices (4) to (6).

The client device 200 is a wearable terminal (hereinafter also simplyreferred to as the wearable terminal 200). The wearable terminal 200includes, for example, at least one of an imaging mechanism and adisplay mechanism and functions as at least one of the devices (1) to(7). In the illustrated example, the wearable terminal 200 is a glassestype terminal and is not limited to this example as long as the wearableterminal 200 has a shape which can be mounted on the body of a user.When the wearable terminal 200 functions as the devices (1) to (3), thewearable terminal 200 includes a camera installed on, for example, aglasses frame as the imaging mechanism. In the wearable terminal 200,the camera can acquire an image of the real space from a position closeto the viewpoint of the user. The acquired image is transmitted to theserver 100 or the other client devices 300 to 700. When the wearableterminal 200 functions as the devices (4) to (7), the wearable terminal200 includes a display installed on a part or all of lenses of glassesas the display mechanism. The wearable terminal 200 causes the displayto display an image captured by the camera.

The client device 300 is a tablet terminal (hereinafter also simplyreferred to as a tablet terminal 300). The tablet terminal 300 includesat least a display mechanism, preferably further includes a manipulationmechanism, and can function as the devices (4) to (7). The tabletterminal 300 may further include an imaging mechanism in addition to thedisplay mechanism and the manipulation mechanism and may function as atleast one of the devices (1) to (3). That is, the tablet terminal 300can function as any device among the devices (1) to (7).

The client device 400 is a mobile phone (smartphone) (hereinafter alsosimply referred to as a mobile phone 400). Since the function of themobile phone 400 in the system 10 is the same as that of the tabletterminal 300, the detailed description thereof will be omitted. Althoughnot illustrated, for example, a device such as a portable game device, aportable music reproduction device, or a digital camera can function inthe same way as the tablet terminal 300 or the mobile phone 400 in thesystem 10 as long as the device includes a communication mechanism, adisplay mechanism, and a manipulation mechanism or an imaging mechanism.

The client device 500 is a laptop personal computer (PC) (hereinafteralso simply referred to as a laptop PC 500). The laptop PC 500 includesa display mechanism and a manipulation mechanism and functions as thedevices (4) to (7). In the illustrated example, since the laptop PC 500is basically fixed for use, the laptop PC 500 is treated as an exampleof a device which does not function as the devices (1) to (3). Althoughnot illustrated, for example, a desktop PC or a television can functionin the same way as the laptop PC 500. The laptop PC 500 includes adisplay as a display mechanism, includes a mouse or a keyboard as amanipulation mechanism, displays images supplied directly from thedevices (1) to (3) or via various devices, and receives variousmanipulations performed on the images by the user. When the laptop PC500 further includes an imaging mechanism such as a camera, the laptopPC 500 can also function as the devices (1) to (3).

The client device 600 is a fixed camera (hereinafter also simplyreferred to as a fixed camera 600). The fixed camera 600 includes animaging mechanism and functions as the devices (1) to (3). In theillustrated example, since the fixed camera 600 is fixed for use anddoes not include a display mechanism, the fixed camera 600 is treated asan example of a device which does not function as the devices (4) to(7). Although not illustrated, for example, when a camera photographingthe front of a screen is installed in a desktop PC or a television orwhen a movable device such as a digital camera is temporarily fixed on atripod, the device can function in the same way as the fixed camera 600.The fixed camera 600 includes a camera as an imaging mechanism and canacquire an image of the real space from a fixed viewpoint (including acase in which the camera swings automatically or according to amanipulation of the user viewing a captured image).

The client device 700 is a projector (hereinafter also simply referredto as a projector 700). The projector 700 includes a projection deviceas a display mechanism and functions as the device (7). In theillustrated example, since the projector 700 does not include an imagingmechanism and does not include a manipulation mechanism receiving aninput of a displayed (projected) image either, the projector 700 istreated as an example of a device which does not function as the devices(1) to (6). The projector 700 displays various images in the real spaceby projecting images onto a screen or the surface of an object using aprojection device. The projector 700 is illustrated as fixed typeprojector, but may be a handheld type projector.

The server 100 functions as at least one of the devices (1) to (7)independently or in cooperation with the client devices 200 to 700. Thatis, the server 100 has a function of acquiring an image of the realspace, performing various kinds of image processing on an obtainedimage, or displaying at least one of the acquired image of the realspace or an image obtained through the image processing.

Through the foregoing functions realized by the server 100 and theclient devices 200 to 700, the user can view an image of the real spacein which a moving object such as any of various life forms such as humanbeings, a self-propelled object propelling itself on the ground,underground, or underwater, or a flying object flying in the air ispresent, and thus the space can be shared between any of the variousmoving objects and the user. In the system according to the embodiment,the user can also freely view an image of the real space in which amoving object is present independently from the moving object byperforming a process to be described in detail below.

The system according to the embodiment has been described above. Asillustrated in FIG. 1, the system 10 according to the embodiment caninclude the device capable of acquiring an image of the real space, thedevice capable of supplying the image of the real space to the user sothat the user can view the image of the real space and receiving variousmanipulations by the user, and the device capable of displaying theimage generated through various manipulations by the user.

The server 100 and the client devices 200 to 700 independently orcooperatively perform various kinds of information processing includingthe above-described image processing performed by the system 10. Theserver 100 and the client devices 200 to 700 independently orcooperatively realize the information processing device to be describedbelow in detail in terms of the entire system 10.

[Device Configuration]

FIG. 2 is an explanatory diagram illustrating a schematic configurationof a device according to the embodiment. As illustrated in FIG. 2, adevice 900 includes a processor 910 and a memory 920. The device 900 canfurther include at least one of a display unit 930, a manipulation unit940, a communication unit 950, an imaging unit 960, and a sensor 970.These constituent elements are mutually connected by a bus 980. Thedevice 900 can realize, for example, a server device configured as theforegoing server 100 and any of the client devices 200 to 700.

The processor 910 is, for example, any of various processors such as acentral processing unit (CPU) or a digital signal processor (DSP) andrealizes various functions, for example, by performing operations suchas calculation or control according to programs stored in the memory920. The processor 910 realizes, for example, a control function of anyentire device of the server 100 and the client devices 200 to 700. Theprocessor 910 performs, for example, various kinds of image processingto be described below or display control to display an image on adisplay screen in the server 100 or the client devices 200 to 700.

The memory 920 is configured to include a storage medium such as asemiconductor memory or a hard disk and stores programs or data used forprocesses by the device 900. The memory 920 may store, for example,captured image data acquired by the imaging unit 960 or sensor dataacquired by the sensor 970. Some of the programs and data to bedescribed in the present specification may be acquired from an externaldata source (for example, a data server, a network storage, or anexternally attached memory) without being stored in the memory 920.

The display unit 930 is installed in, for example, a client includingthe above-described display mechanism. The display unit 930 can be, forexample, a display according to the shape of the device 900. Forexample, in terms of the foregoing example, the wearable terminal 200can include a display which has a shape corresponding to a lens ofglasses or a shape corresponding to a display region of a head-mounteddisplay. The tablet terminal 300, the mobile phone 400, or the laptop PC500 can include a flat type display installed in each casing.Alternatively, the display unit 930 may be a projection device thatprojects an image to an object. In the foregoing example, the projector700 can include a projection device as a display unit.

The manipulation unit 940 is installed in, for example, a clientincluding the above-described manipulation mechanism. The manipulationunit 940 is configured by combining a keyboard, a button, a switch, orthe like with a pointing device such as a touch sensor (which configuresa touch panel along with a display) installed on the display, a touchpad, or a mouse, as necessary. For example, the manipulation unit 940receives a manipulation of the user specifying a position inside animage displayed on the display unit 930 by the pointing device andinputting any information at the position with the keyboard, the button,the switch, or the like. Alternatively, the manipulation unit 940 mayreceive a manipulation of the user specifying a position inside an imagedisplayed on the display unit 930 by the pointing device and inputtingany information at the position with the pointing device.

The communication unit 950 is a communication interface which relayscommunication between the device 900 and another device. Thecommunication unit 950 supports any wireless communication protocol orany wired communication protocol and establishes communicationconnection with another device.

The imaging unit 960 is a camera module that captures an image. Theimaging unit 960 images the real space using an image sensor such as acharge coupled device (CCD) or a complementary metal oxide semiconductor(CMOS) and generates a captured image. A series of captured imagesgenerated by the imaging unit 960 configures a video. The imaging unit960 may not necessarily be included as a part of the device 900. Forexample, an imaging device connected to the device 900 in a wireless orwired manner may be treated as the imaging unit 960. The imaging unit960 may include a depth sensor that measures a distance between theimaging unit 960 and a subject for each pixel. Depth data output fromthe depth sensor can be used to recognize an environment of an imageobtained by imaging the real space, as will be described below.

The sensor 970 can include various sensors such as a positioning sensor,an acceleration sensor, and a gyro sensor. A measured result obtained bythe sensor 970 may be used for various purposes to support therecognition of the environment of the image obtained by imaging the realspace, acquire data specific to a geographical location, or detect auser input. The sensor 970 can be installed in a device including theimaging unit 960 (in the foregoing example, the wearable terminal 200,the tablet terminal 300, the mobile phone 400, or the fixed camera 600,or the like).

<Configuration of Information Processing Device>

Next, the configuration of the information processing device accordingto the embodiment realized independently or cooperatively by the server100 and the client devices 200 to 700 described above in terms of theentire system 10 will be described in detail focusing on mainly thefunctions with reference to FIGS. 3 to 10.

Here, classification of a captured image handled by an informationprocessing device 10 according to the embodiment is not particularlylimited, but may be a still image or a moving image.

A captured image handled by an information processing device 1000according to the embodiment is preferably a captured image obtained byimaging a range of the real space as widely as possible. Accordingly, animaging device used to image the real space is preferably a camera onwhich as wide an angle lens as possible is mounted and is morepreferably, for example, an omnidirectional camera schematicallyillustrated in FIG. 3.

FIG. 3 schematically illustrates a configuration in which anomnidirectional camera imaging the real space is realized as thewearable terminal 200. In the wearable terminal 200 illustrated in FIG.3, cameras on which angle lenses that are as wide as possible aremounted are installed in a circular form to surround the circumferenceof a human head which is an example of a moving object. Even when thecameras are installed in the circumference of the human head, it isdifficult to obtain an image in a zenith direction. Therefore, a camerais also installed on the top of the head in FIG. 3. In the wearableterminal 200, various sensors such as a positioning sensor, anacceleration sensor, and a gyro sensor are installed. Informationregarding a visual line of the imaging device (in other words, theposture of an imaging device) output from the sensor is output to aninformation processing device to be described below and is used asposture information of the imaging device which is information regardingthe posture of the imaging device in the information processing device.

In the example illustrated in FIG. 3, the case in which the cameras aredisposed in the circular form to obtain an omnidirectional image isillustrated. However, when it is not necessary for an image handled inthe information processing device 1000 to be an omnidirectional image,the cameras may not be installed in the circular form and the camerasmay be installed at least at parts of the human head. The number ofcameras used to implement the wearable terminal 200 illustrated in FIG.3 is not limited, but the number of cameras may be appropriately set sothat an image with a certain wide range can be obtained.

In FIG. 3, the case in which the moving object is a human being isillustrated, but the moving object is not limited to a human being. Themoving object may be an animal other than a human being on which thewearable terminal 200 is mounted or may be a self-propelled object suchas a robot or a flying object on which cameras are mounted.

The information processing device 1000 that performs various kinds ofimage processing on a captured image captured by the imaging deviceexemplified in FIG. 3 is a device that performs control such that adisplay image generated based on image information generated throughimaging by an imaging device mounted on a moving object moving in aspace, imaging-device posture information which is information regardingthe posture of the imaging device, and user view information which isobtained from the user manipulation device manipulated by the user andspecifies a region that the user desires to view is displayed in adisplay region viewed by the user. The imaging-device postureinformation may be, for example, information regarding rotation of theimaging device. The user view information may be, for example,information specifying a display field angle that the user desires toview in the omnidirectional image captured by the imaging device.

As illustrated in FIG. 4A, for example, the information processingdevice 1000 includes at least a display control unit 1050 which is anexample of a control unit. As illustrated in FIG. 4B, the informationprocessing device 1000 according to the embodiment may further includeat least one of an image generation unit 1010, an image selection unit1020, an image correction unit 1030, a moving-object visual lineinformation generation unit 1040, a data acquisition unit 1060, a datasupply unit 1070, and a storage unit 1080 in addition to the displaycontrol unit 1050. Here, the processing units illustrated in FIGS. 4Aand 4B may be realized in any one of the server 100 and the clientdevices 200 to 700 or may be distributed to the plurality of devices tobe realized.

In the following description, a case in which the information processingdevice 1000 performs display control on the display image generatedbased on the imaging-device posture information, the user viewinformation, and the captured image captured by the imaging device willbe described. It is needless to say that the information processingdevice 1000 may perform the display control as follows based on the userview information and generated image (for example, corrected imageobtained by performing correction of the posture of the imaging deviceon the captured image in advance) generated based on the captured imageand the imaging-device posture information by the imaging device ordevices other than the imaging device and the information processingdevice.

The image generation unit 1010 generates circumferential captured imageswhich are captured in the circumference of a position at which a movingobject moving in a space is present using captured images captured bythe imaging device mounted on the moving object. A process of generatingthe circumferential captured image by the image generation unit 1010 isperformed continuously in real time, for example, when the capturedimages are output from the imaging device illustrated in FIG. 3.

Here, when the captured images to be used to generate thecircumferential captured image are captured by the omnidirectionalcameras exemplified in FIG. 3, the circumferential captured imagegenerated by integrating the captured images by the image generationunit 1010 is an omnidirectional captured image (spherical image)illustrated in FIG. 5. A scheme of generating a circumferential capturedimage from a plurality of captured images captured by a plurality ofcameras is not particularly limited, but a known scheme may be applied.

The image generation unit 1010 may generate a rectangular imageequivalent to the spherical image and illustrated in FIG. 6 as thecircumferential captured image, rather than an omnidirectional capturedimage (spherical image) illustrated in FIG. 5. The rectangular imageequivalent to the spherical image can be generated, for example, byconverting the spherical image in accordance with a known method such asequidistant cylindrical projection. By using the rectangular imageillustrated in FIG. 6 as the circumferential captured image rather thanthe spherical image illustrated in FIG. 5, it is possible to performvarious kinds of image processing more simply.

The image selection unit 1020 selects the captured image correspondingto the user view information as a user view image among thecircumferential captured images based on the circumferential capturedimage generated by the image generation unit 1010 and user viewinformation which is obtained from a user manipulation devicemanipulated by the user and indicates a space that the user desires toview. The user view image selected by the image selection unit 1020 issupplied to the user manipulation device (for example, the wearableterminal 200 such as a head-mounted display mounted on a different userfrom a moving object in the examples illustrated in FIGS. 5 and 6)manipulated by the user to be supplied for the user to view.Accordingly, the user manipulating the user manipulation device canshare a certain space with the moving object moving in the space andselect a position that the he or she desires to view in the spaceindependently from the moving object. As a result, in the space in whichthe moving object is present, the user can freely select an image at adifferent position from the position viewed by the moving object.

The generation process for the circumferential captured images and theimage selection process from the circumferential captured images can beperformed at lower calculation cost than in a space recombinationtechnology in which a process at high calculation cost, such ascollation of feature points between images, is frequently used.Accordingly, it is possible to realize downsizing and lightening of theinformation processing device 1000 capable of performing the process.

Here, the user view information set by the user manipulation device isgenerated when the user manipulates various input mechanisms such as atouch pad, a keyboard, and a mouse installed in the user manipulationdevice, and is transferred to the image selection unit 1020. When theuser manipulation device is the wearable terminal 200 illustrated inFIGS. 5 and 6, the user view information may be generated by varioussensors such as the positioning sensor, the acceleration sensor, and thegyro sensor installed on the wearable terminal 200 automaticallydetecting a behavior of the user (for example, a visual line directionof the user). The user view information may be generated when the userperforms audio input or gesture input to the user manipulation device.

In this way, the information processing device 1000 according to theembodiment includes the image generation unit 1010 and the imageselection unit 1020, and thus supplies an image (so-called first-personviewpoint image) of the space viewed by the moving object (morespecifically, the imaging device) to the user in real time. Here, in thefirst-person viewpoint image, considerable screen shake occurs in somecases since the moving object (more specifically, the imaging device)looks in the circumference of a position at which the moving object ispresent. When the user views the considerable screen shake, the user mayfeel “nausea” (motion sickness) caused in some cases due to seeing animage with considerable shaking. Accordingly, the information processingdevice 1000 according to the embodiment preferably has a correctionfunction of correcting the foregoing rotation movement of the imagingdevice.

The image correction unit 1030 is a processing unit that corrects achange in the image accompanying the above-described rotation movementof the imaging device based on the imaging-device posture information.When the visual line direction of the imaging device is changed withouta change in the position of the imaging device (that is, a rotationmovement occurs in the imaging device), the image correction unit 1030performs correction on the circumferential captured image such that achange in the circumferential captured image accompanying the change inthe visual line direction of the imaging device is suppressed.

More specifically, the image correction unit 1030 uses moving-objectvisual line information to perform correction such that thecircumferential captured image after the change in the visual linedirection of the imaging device is reversely rotated according to themagnitude of a rotation angle accompanying the change in the visual linedirection of the imaging device. Hereinafter, this correction processwill be described with reference to FIG. 7.

As illustrated in FIG. 7, a spherical image A is assumed to be generatedat a certain time in accordance with captured data from the movingobject (human being) wearing the wearable terminal 200 which includesthe sensors and the omnidirectional cameras. Thereafter, it is assumedthat a rotation movement of the imaging device occurs, a change in thevisual line direction occurs, and a spherical image B is accordinglygenerated. In this case, the image correction unit 1030 extracts arotation component with reference to sensor information output from thewearable terminal 200 and specifies the magnitude of a rotation angleaccompanying the change in the visual line direction of the imagingdevice. Subsequently, the image correction unit 1030 performs thecorrection on the spherical image B such that the image is reverselyrotated according to the obtained magnitude of the rotation angle andgenerates a spherical image C in which the rotation component iscanceled from the spherical image B. Accordingly, as the cancellationresult of the rotation component, the spherical image C becomes an imageviewed in substantially the same direction as the spherical image A.

By performing the image correction process in this way, it is possibleto suppress the considerable screen shake caused due to the rotationmovement of the imaging device, and thus it is possible to prevent“nausea” (motion sickness) of the user from occurring.

The image correction unit 1030 may perform a process of correcting therotation movement so that local feature amounts match before and afterthe change in the visual line direction accompanying the rotationmovement of the imaging device. FIG. 7 illustrates the case in which therotation correction is performed using the output from the sensorinstalled in the moving object. However, the correction process for therotation movement may be performed focusing on the local feature amountin the spherical image A and the local feature amount in the sphericalimage B.

For example, the image correction unit 1030 extracts the local featureamount (for example, the positions of feature points) in the sphericalimage A and the local feature amount in the spherical image B andperforms a process of collating the local feature amounts. Further, theimage correction unit 1030 may perform the correction so that the localfeature amounts match before and after the change in the visual linedirection of the imaging device. In this case, the image correction unit1030 may extract the rotation component for matching two local featureamounts and perform the correction on the spherical image B to reverselyrotate the image according to the obtained magnitude of the rotationangle.

Here, the local feature amounts focused on by the image correction unit1030 are not particularly limited, but known local feature amounts canbe used. For example, Scale invariant Feature Transform (SIFT) can beexemplified as the local feature amount.

As illustrated in FIG. 8, the image correction unit 1030 may use theimage correction process based on the output from the sensor mounted onthe moving object and the image correction process based on the localfeature amounts. Accordingly, the image correction unit 1030 can cancelthe rotation component more precisely.

The image correction unit 1030 may control the degree to which thecorrection is performed according to correction application informationindicating the application degree of the correction which is obtainedfrom the user manipulation device. Accordingly, the image correctionunit 1030 can completely cancel the rotation component through theforegoing rotation component correction or may not correct the rotationcomponent, or can perform correction to an extent at which the rotationcomponent is not completely canceled. The image correction unit 1030 canalso perform image control to gradually follow the rotation movement ofthe imaging device by performing the correction to an extent at whichthe rotation component is not completely canceled.

The rotation movement generated in the imaging device can be expressed,for example, using rotation coordinate axes which are defined mutuallyindependently, such as a yaw axis, a pitch axis, and a roll axis.Therefore, the image correction unit 1030 may independently control thedegree to which the above-described rotation correction is performed foreach of the rotation coordinate axes, for example, as illustrated inFIG. 9.

The moving-object visual line information generation unit 1040 generatesvisual line information indicating the visual field or the direction(position) of the visual line of the imaging device based on theimaging-device posture information. The visual line information can begenerated in accordance with a known direction, for example, usingoutput information (that is, the imaging-device posture information)from the various sensors mounted on the moving object. The visual lineinformation can be supplied to the user along with the circumferentialcaptured image generated by the image generation unit 1010, so that anobject indicating the visual field or the direction (position) of thevisual line of the imaging device can be displayed in the user viewimage supplied to the user manipulation device. As a result, the usercan comprehend the visual line direction of the imaging device at alltimes while viewing the circumferential captured image in any directiondifferent from the visual field or the visual line direction (position)of the imaging device.

The display control unit 1050 controls display content of a displaydevice such as a display installed in the information processing device1000 or outside of the information processing device 1000. Specifically,the display control unit 1050 performs control such that a display imagewhich is generated based on the image information generated through theimaging by the imaging device mounted on the moving object moving in thespace, the imaging-device posture information which is informationregarding the posture of the imaging device, and the user viewinformation which is obtained from the user manipulation devicemanipulated by the user and specifies the region that the user desiresto view is displayed in a display region viewed by the user. The displaycontrol unit 1050 can display the object indicating the visual field andthe visual line direction of the imaging device inside the user viewimage by performing display control on the display screen of the usermanipulation device, for example, as illustrated in FIG. 10.Accordingly, the user can comprehend the visual line direction of themoving object at all times while selecting the visual line directionindependently from the moving object.

The data acquisition unit 1060 acquires captured image data output fromthe imaging device mounted on the moving object or visual-line-relateddata including sensor output (that is, the imaging-device postureinformation) regarding the visual line direction of the imaging device,or acquires data regarding a user manipulation output from the usermanipulation device. Various kinds of data acquired from the variousdevices by the data acquisition unit 1060 can be used appropriately byeach processing unit included in the information processing device 1000.

The data supply unit 1070 supplies various kinds of data generated bythe information processing device 1000 (for example, captured image datasuch as the circumferential captured image or the user view image orvisual-line-related data such as the visual line direction of theimaging device) to a device installed outside of the informationprocessing device 1000. Accordingly, the device installed outside of theinformation processing device 1000 can also use the various kinds ofinformation generated by the information processing device 1000.

The storage unit 1080 may appropriately record various databases usedfor processes of the image generation unit 1010, the image selectionunit 1020, the image correction unit 1030, the moving-object visual lineinformation generation unit 1050, the display control unit 1050, thedata acquisition unit 1060, and the data supply unit 1070, variousprograms including applications used for various calculation processesperformed by these processing units, and various parameters necessarilystored when certain processes are performed or courses of interimprocesses.

The storage unit 1080 can be freely accessed by each processing unitsuch as the image generation unit 1010, the image selection unit 1020,the image correction unit 1030, the moving-object visual lineinformation generation unit 1050, the display control unit 1050, thedata acquisition unit 1060, and the data supply unit 1070, so that datacan be written or read.

The example of the function of the information processing device 1000according to the embodiment has been described. The foregoingconstituent elements may be configured using general members or circuitsor may be configured by hardware specialized for the functions of theconstituent elements. All of the functions of the constituent elementsmay be performed by a CPU or the like. Accordingly, the configurationsto be used can be changed appropriately according to technology levelswhenever the embodiment is realized.

A computer program for realizing each function of the informationprocessing device according to the above-described embodiment can becreated to be mounted on a personal computer or the like. Acomputer-readable recording medium in which such a computer program isstored can also be supplied. Examples of the recording medium include amagnetic disk, an optical disc, a magneto-optical disc, and a flashmemory. The computer program may be delivered via, for example, anetwork without using a recording medium.

The image generation unit 1010, the image selection unit 1020, the imagecorrection unit 1030, the moving-object visual line informationgeneration unit 1050, the data acquisition unit 1060, the data supplyunit 1070, and the storage unit 1080 illustrated in FIG. 4B may also bemounted on another device such as a computer capable of mutuallycommunicating with the information processing device 1000, so that theforegoing functions can be realized in cooperation with the informationprocessing device 1000 and another device.

<Flow of Information Processing Method>

Next, the flow of an information processing method performed by theinformation processing device 1000 according to the embodiment will bedescribed in brief with reference to FIG. 11.

In the information processing method according to the embodiment, thecaptured image data is first acquired from the cameras mounted on themoving object (step S101).

Thereafter, the image generation unit 1010 of the information processingdevice 1000 generates the circumferential captured images for, forexample, the spherical image and the rectangular image obtained byconverting the spherical image into a rectangular form based on theacquired captured image data (step S103).

At this time, the image correction unit 1030 of the informationprocessing device 1000 performs the foregoing correction process on thegenerated circumferential captured images, as necessary (step S105).

Thereafter, the image selection unit 1020 of the information processingdevice 1000 selects an image corresponding to the user view information(that is, the user view image) among the circumferential captured imagesaccording to the user view information acquired from the usermanipulation device (step S107).

Subsequently, the display control unit 1050 of the informationprocessing device 1000 controls display of the selected image on thedisplay screen of the user manipulation device (step S109). Accordingly,the user using the user manipulation device can share the image in thespace in which the moving object is present with the moving object.

Even when the process is performed using a generated image generatedbased on the captured image and the imaging-device posture informationinstead of the captured image acquired from the imaging device, theprocess described above can be performed.

The flow of the information processing method according to theembodiment has been described in brief above with reference to FIG. 11.

<Example of Display Control Process>

Next, a display control process in the display control unit 1050 will bedescribed specifically with reference to FIGS. 12 to 17 exemplifying thedisplay image generated by the display control unit 1050 of theinformation processing device 1000 according to the embodiment.

FIGS. 12 to 16 are explanatory diagrams for describing the displaycontrol process according to the embodiment. FIG. 17 is a flowchartillustrating a flow example of the display control process according tothe embodiment.

As described with reference to FIGS. 7 and 8, the information processingdevice 1000 according to the embodiment can extract the rotationcomponent by focusing on a change between frames of the spherical imagesgenerated from the image information continuously output from theimaging device or the rectangular images (that is, the circumferentialcaptured image) which are equivalent to the spherical images and arebased on equidistant cylindrical projection.

Here, a rotation movement Q_(1,2) generated between a circumferentialcaptured image F₁ corresponding to a frame 1 and a circumferentialcaptured image F₂ corresponding to a frame 2 can be specified through aknown estimation process or the like performed focusing on locations inthe circumferential captured image F₂ of the positions of feature pointsin the circumferential captured image F₁. By continuously performing therotation movement specifying process up to a rotation movement Q_(N-1,N)generated between a circumferential captured image F_(N-1) correspondingto a frame (N−1) and a circumferential captured image F_(N)corresponding to a frame N to take a product of the obtained rotationmovements, it is possible to specify a rotation movement Q_(1,N) fromthe frame 1 to the frame N as in the following expression 101.Q _(1,N) =Q _(1,2) ×Q _(2,3) × . . . ×Q _(N-1,N)  (expression 101)

The rotation movement Q_(1,N) obtained in this way can be said to beinformation regarding rotation accompanying a change in the visual linedirection of the imaging device (that is, rotation information). Therotation information can be handled as information indicating atrajectory of the rotation movement generated between the frame 1 to theframe N. The display control unit 1050 visualizes the rotation movementQ_(1,N), and thus can supply the user with the trajectory of therotation movement generated between the frame 1 to the frame N (in otherwords, posture information for visualizing a change in the posture ofthe imaging device).

Here, as coordinate systems available when the display control unit 1050superimposes the posture information on the display image (that is, thecircumferential captured image) generated by the image generation unit1010, there are the following two coordinate systems:

(a) a coordinate system fixed to the space in which the imaging deviceis present (an absolute coordinate system: hereinafter also referred toas a coordinate system A); and

(b) a coordinate system fixed to the imaging device (a relativecoordinate system: hereinafter also referred to as a coordinate systemB).

The display control unit 1050 according to the embodiment appropriatelydisplays the circumferential captured image in the space using thecoordinate system A between the foregoing two coordinate systems andsuperimposes various objects or an image corresponding to the postureinformation on the circumferential captured image.

The display control unit 1050 according to the embodiment may furthersuperimpose various objects or an image corresponding to the postureinformation on the circumferential captured image subjected to thecorrection such that the change in the circumferential captured imageaccompanying the change in the visual line direction of the imagingdevice is suppressed.

The user view image selected from the circumferential captured images bythe image selection unit 1020 corresponds to an image seen when a partof the circumferential captured image mapped to the surface of theentire sphere in the coordinate system A schematically illustrated inFIG. 12 is viewed from any point located inside the sphere. Accordingly,apart from the coordinate system A, the coordinate system B is definedat any point located inside the sphere. As illustrated in FIG. 12, thecoordinate systems available in the embodiment are preferably rotationcoordinate systems that express any position on the surface of thesphere using two rotation angles.

When various annotations such as text data, image data, and audio dataare requested to be added to specific positions of the circumferentialcaptured images from the user manipulation device, the display controlunit 1050 preferably associates the various annotations withcorrespondence spots of positions designated from the user manipulationdevice in the coordinate system A. The display control unit 1050 canalso display various objects such as icons or images indicating theannotations at correspondence spots (the correspondence spots in thecoordinate system A) of the circumferential captured images according toa position designated from the user manipulation device.

Here, when the information included in the posture information isvisualized, at least the following two methods can be used as methodsadopted by the display control unit 1050. The display control unit 1050sets or changes the adopted visualization method based on a usermanipulation of at least one of a manipulator of the imaging device anda manipulator of the user manipulation device.

(A) A motion of the coordinate system A is fixed and a motion of thecoordinate system B is Changed According to the Posture Information.

In this case, display is realized so that the circumferential capturedimage displayed in the display region of the display device such as theuser manipulation device is changed with the change in the posture ofthe imaging device. The various objects or the images corresponding tothe posture information are displayed so that the various objects or theimages are fixed to the display region even when the posture of theimaging device is changed.

(B) A motion of the coordinate system A is changed according to theposture information and a motion of the coordinate system B is fixed.

In this case, even when the posture of the imaging device is changed,the display is realized so that the circumferential captured imagedisplayed in the display region of the display device such as the usermanipulation device is not changed or the change in the imageaccompanying the change in the posture of the imaging device is small.The various objects or the images corresponding to the postureinformation are displayed in the display region so that the variousobjects or the images are changed (virtually rotated) with the change inthe posture of the imaging device.

Here, the above-described image correction process performed by theimage correction unit 1030 corresponds to a process of fixing a motionof the coordinate system B in the foregoing visualization method of (B).

By visualizing the posture information in accordance with the foregoingvisualization method of (A), the circumferential captured image fixed tothe coordinate system A and the direction of the coordinate system B ischanged according to the posture information in the fixed state. Toconvey the change in the motion of the coordinate system B to the userof the user manipulation device more clearly, the display control unit1050 may superimpose an object indicating the coordinate axes of thecoordinate system B (for example, coordinate axes defined with angles ofthe latitude direction and the longitude direction) as an objectindicating the posture information on the circumferential captured imageand may rotate the coordinate axes according to the posture information,for example, as illustrated in FIG. 12. In order to efficiently conveythe change in the motion of the coordinate system B to the user of theuser manipulation device, the display control unit 1050 may superimposethe motion corresponding to the change in the posture information as atrajectory on the circumferential captured image, for example, asillustrated in FIG. 13.

By visualizing the posture information in accordance with the foregoingvisualization method of (B), the direction of the coordinate system B isfixed and the circumferential captured image fixed to the coordinatesystem A is changed according to the posture information in the fixedstate. In this case, since the circumferential captured image is rotatedaccording to the posture information, the user of the user manipulationdevice can easily comprehend the change in the posture of the imagingdevice. Here, the display control unit 1050 may superimpose an objectindicating the coordinate axes of the coordinate system A (for example,coordinate axes defined with angles of the latitude direction and thelongitude direction) on the circumferential captured image and may alsorotate the coordinate axes according to the rotation of thecircumferential captured image, for example, as illustrated in FIG. 12.In order to efficiently convey the change in the motion of thecoordinate system A to the user of the user manipulation device, thedisplay control unit 1050 may superimpose the motion corresponding tothe change in the posture information as a trajectory on thecircumferential captured image, for example, as illustrated in FIG. 13.

When the display control unit 1050 visualizes the rotation of the twotypes of coordinate systems described above, the display control unit1050 can superimpose at least one of an object that is rotated with arotation movement accompanying the change in the visual line directionof the imaging device and an object that is not rotated on the generateddisplay image (that is, the user view image). That is, the displaycontrol unit 1050 rotates the object indicating the coordinate axes ofthe coordinate system with the rotation movement, but may not rotate anobject, for example, a numerical value or a letter given on thecoordinate axes illustrated in FIG. 12, which would be difficult tocomprehend if the object were rotated with the rotation movement.Accordingly, the position of the object which would be difficult tocomprehend if the object were rotated with the rotation movement ismoved with the rotation movement, but the posture of the object can beconstant from the viewpoint. As a result, it is possible for the user tocomprehend the object more easily.

Here, the specific examples of the various objects superimposed on theuser view image by the display control unit 1050 are not particularlylimited, but any object can be used. It is needless to say that thevisual line information illustrated in FIG. 10 may be superimposed onthe user view image.

The display control unit 1050 preferably decides or changes the settingregarding which coordinate axes are displayed between the coordinateaxes of the coordinate system A and the coordinate axes of thecoordinate system B and which coordinate axes are rotated, based on auser manipulation of at least one of the manipulator of the imagingdevice and the manipulator of the user manipulation device.

When the rotation axes of the coordinate system match the rotation axesof the rotation movement described in the posture information (that is,the rotation movement of the imaging device), it is difficult for theuser of the user manipulation device to comprehend the change in theposture of the imaging device in some cases. Accordingly, when thedisplay control unit 1050 visualizes the change in the posture of theimaging device using the rotation information, the display control unit1050 preferably generates the display image by virtually viewing thespace at a different position (for example, a position O from the centerof the coordinate system (the coordinate system A) fixed to the spacetranslated from a center C of the coordinate system A in FIG. 12backward in the visual line direction of the imaging device).Accordingly, the user of the user manipulation device can view a displayimage just as the display image is generated from a fixed camerainstalled visually at a different position from the position of theimaging device. As a result, the user of the user manipulation devicecan more easily comprehend the change in the posture of the imagingdevice. The display control unit 1050 can set or change a referenceposition (the position O in FIG. 12) in the foregoing visualizationbased on a user manipulation of at least one of the manipulator of theimaging device and the manipulator of the user manipulation device.

In order to more efficiently convey the change in the postureinformation to the user of the user manipulation device, the displaycontrol unit 1050 may control at least one of a reproduction speed and adisplay field angle at the time of the display of the display image inthe display region viewed by the user according to the postureinformation. The display control unit 1050 can more efficiently conveythe change in the posture information to the user of the usermanipulation device by performing display control such as a reduction ina reproduction speed, for example, at a time point at which a rotationamount based on the posture information is large.

The display control unit 1050 may generate a display image in a case ofvirtual viewing of the space at an arbitrary position designated fromthe user manipulation device, centering on the designated arbitraryposition, and may supply the display image to the user manipulationdevice.

[Specific Example of Display Image]

Hereinafter, an example of a display image transferred to the usermanipulation device through the display control process performed by thedisplay control unit 1050 according to the embodiment will be describedin brief with reference to FIGS. 14 to 16.

FIGS. 14 and 15 to be described below illustrate an example of a case inwhich images captured by the wearable terminal 200 are supplied toanother user manipulating the user manipulation device when the userwearing the wearable terminal 200 on which the imaging deviceillustrated in FIG. 3 is mounted exercises on a horizontal bar which isone of the gymnastics events in a gymnasium.

FIG. 14 illustrates examples of display images when the foregoingvisualization method of (A) is adopted. As apparent from FIG. 14, it canbe understood that when the user wearing the wearable terminal 200continues with the horizontal bar event, circumferential images arechanged and the direction of an object indicating the coordinate axes ofthe coordinate system A is changed moment by moment with the change. InFIG. 14, a trajectory indicating the posture information of the wearableterminal 200 is superimposed on the user view image.

On the other hand, FIG. 15 illustrates an example of a display imagewhen the foregoing visualization method of (B) is adopted. As apparentfrom FIG. 15, it can be understood that even when the user wearing thewearable terminal 200 continues with the horizontal bar event,circumferential images are not changed and the coordinate axes of thecoordinate system B displayed in the superimposition manner are changedmoment by moment.

By performing the display control process according to the embodiment inthis way, it is possible to visualize the change in the posture of theimaging device using the obtained posture information (the rotationinformation).

The display control process according to the embodiment can be appliednot only to the spherical images exemplified in FIGS. 14 and 15 but alsoto a rectangular image equivalent to the spherical image in accordancewith equidistant cylindrical projection, as illustrated in FIG. 16.

[Flow of Display Control Process]

Next, a flow example of the display control process according to theembodiment will be described in brief with reference to FIG. 17.

In the display control process according to the embodiment, capturedimage data is first acquired from the cameras mounted on the movingobject (step S151).

Thereafter, the image generation unit 1010 of the information processingdevice 1000 generates the circumferential captured images such as thespherical images or the rectangular images obtained by converting thespherical images into a rectangular form based on the acquired capturedimage data (step S153).

Subsequently, the image correction unit 1030 of the image generationunit 1010 performs the rotation analysis process described above usingthe generated circumferential captured images (step S155).

Thereafter, the display control unit 1050 of the information processingdevice 1000 disposes the generated circumferential captured images orvarious graphic objects on a celestial sphere based on the coordinatesystem A (step S157).

Subsequently, the image selection unit 1020 of the informationprocessing device 1000 generates an image corresponding to the user viewinformation (that is, the user view image) from the circumferentialcaptured images on which the various graphic objects are superimposedaccording to the user view information acquired from the usermanipulation device (step S159).

The display control unit 1050 of the information processing device 1000controls the display of the image selected by the image selection unit1020 on the display screen of the user manipulation device so that theuser using the user manipulation device can share the image in the spacein which the moving object is present with the moving object.

The flow of the display control process according to the embodiment hasbeen described above in brief with reference to FIG. 17.

<Conclusion>

In this way, in the information processing device and the informationprocessing method according to the embodiment, a video surrounding themoving object can be observed as the circumferential captured images inreal time, and thus the user can obtain the sense of presence just as ifthe user were in the same location as the moving object. By performingthe foregoing correction process, the shake of the images caused due tothe rotation movement of the moving object is suppressed. Therefore, theuser can avoid motion sickness (video sickness) caused due to an abruptchange in the images.

For such a configuration, there are a unilateral case (in whichinformation flows unilaterally from the moving object to the user) and abidirectional case (in which information is delivered from the user tothe moving object through audio and other means). For the former case,for example, when the moving object is assumed to be a sports player,the user is a person watching him or her play, and thus it is possibleto realize a sports broadcast in which there is the sense of presence.In this case, the number of users is not limited to one, but tens ofthousands of people similar to broadcast listeners can also beconsidered. On the other hand, for the latter case, a use in which auser gives certain guidance or instructions to a moving object whilesharing the visual field of the moving object is assumed. For example, ause in which, when a moving object (person) is cooking, a user givescooking instructions can be assumed. Even in this case, the number ofusers is not limited to one, but a relatively smaller number of users ismore realistic.

The information processing device and the information processing methodaccording to the first embodiment of the present disclosure will bedescribed in detail.

Second Embodiment Configuration of Information Processing Device

Next, a second embodiment of the present disclosure will be described.An information processing device 1100 according to the embodiment is adevice that performs various kinds of information processing on acaptured image captured by an imaging device mounted on a moving objectwhich moves so that a user can watch an omnidirectional captured imagein the circumference of the moving object more naturally

Specifically, as in the first embodiment, the information processingdevice 1100 according to the embodiment supplies an image (so-calledfirst-person viewpoint image) of a space viewed by a moving object (morespecifically, an imaging device) to a user in real time. Here, in thefirst-person viewpoint image, shake caused due to a motion of the imagedmoving object is included. Therefore, when the user views thefirst-person viewpoint image, the user may feel “nausea” (motionsickness) due to inconsistency of the motion causing the shake includedin the first-person viewpoint image and a motion of the body of theuser.

Accordingly, in order to reduce the “nausea” felt by the user, a processof extracting the shake of the first-person viewpoint image caused dueto the motion of the moving object (more specifically, the imagingdevice) as a rotation component of the image and correcting the image byreverse rotation of the rotation component is performed. In theforegoing process, however, rotation of the imaging device may also becorrected, for example, when the moving object changes its movementdirection. For this reason, when the user watches the first-personviewpoint image of the moving object which moves while changing itsmovement direction, the first-person viewpoint image is normallyoriented in a constant direction, and thus gives an unnatural impressionto the user in some cases.

The information processing device 1100 according to the embodimentsupplies a more natural first-person viewpoint image to the user bycontrolling display of the first-person viewpoint image based on amovement direction of the moving object.

The hardware configurations of a system and the information processingdevice according to the embodiment are substantially the same as theconfigurations illustrated in FIGS. 1 and 2. A captured image handled bythe information processing device 1100 according to the embodiment ispreferably, for example, an image which is captured by omnidirectionalcameras mounted on the wearable terminal 200 illustrated in FIG. 3 andis obtained by imaging a range of the real space as wide as possible(for example, an omnidirectional captured image).

Hereinafter, the specific functional configuration of the informationprocessing device 1100 according to the above-described embodiment willbe described with reference to FIG. 18. Processing units illustrated inFIG. 18 may be realized by any one of the server 100 and the clientdevices 200 to 700 in FIG. 1 or may be distributed to the plurality ofdevices to be realized.

As illustrated in FIG. 18, the information processing device 1100according to the embodiment includes an image generation unit 1110, animage selection unit 1120, an image correction unit 1130, a displaycontrol unit 1150, a data acquisition unit 1160, a data supply unit1170, a storage unit 1180, and a direction control unit 1190.

Here, since the image generation unit 1110 is substantially the same asthe image generation unit 1010, the image selection unit 1120 issubstantially the same as the image selection unit 1020, the displaycontrol unit 1150 is substantially the same as the display control unit1050, the data acquisition unit 1160 is substantially the same as thedata acquisition unit 1160, the data supply unit 1170 is substantiallythe same as the data acquisition unit 1070, and the storage unit 1180 issubstantially the same as the storage unit 1080, the detaileddescription thereof will be omitted here. Hereinafter, the imagecorrection unit 1130 and the direction control unit 1190 characteristicof the embodiment will be described.

The image correction unit 1130 is a processing unit that corrects achange in an image accompanying rotation of the imaging device byperforming reverse rotation to the rotation of the moving object (morespecifically, the imaging device) on a circumferential captured image.The image correction unit 1130 may detect the rotation of the imagingdevice by various sensors such as an acceleration sensor and aninclination sensor and correct a change in the image using the detectedrotation. The image correction unit 1130 may estimate the rotation ofthe imaging device from the circumferential captured images captured bythe imaging device and correct the change in the image using theestimated rotation of the imaging device.

The image correction unit 1130 preferably estimates the rotation of theimaging device from the circumferential captured image captured by theimaging device and corrects the change in the image accompanying theestimated rotation of the imaging device. In this case, since therotation of the imaging device and the rotation correction of thecircumferential captured image are easily synchronized andcorrespondence to higher-speed rotation than when various sensors areused is possible, this is more preferable.

Hereinafter, a method in which the image correction unit 1130 estimatesthe rotation of the imaging device from the circumferential capturedimage and corrects the change in the image based on the estimatedrotation of the imaging device will be described more specifically withreference to FIG. 19.

As illustrated in FIG. 19, a spherical image A is assumed to begenerated at a certain time (time t) in accordance with imaged data froma moving object (human being) wearing the wearable terminal 200including the omnidirectional cameras. Thereafter (at time t+1), it isassumed that a rotation movement is generated in the moving object (morespecifically, the imaging device) and a spherical image B is generated.

In this case, the image correction unit 1130 extracts local featureamounts U from the spherical images A and B and calculates changeamounts F of the local feature amounts. Specifically, the imagecorrection unit 1130 extracts the positions of a plurality (for example,n=1000) of image feature points as the local feature amounts U from thespherical image A and the spherical image B.

Here, the plurality of image feature points are preferably extracted atintervals that are as uniform as possible in the entire sphericalimages. Since distortion of the images tends to increase inhigh-latitude portions of the spherical images, the image feature pointsare preferably not extracted from the high-latitude portions.

Next, the local feature amounts U extracted in the spherical images Aand B are compared to calculate the change amounts F of the localfeature amounts at the time of the change from the spherical image A tothe spherical image B. Of the change amounts F of the local featureamounts, the change amounts equal to or greater than a predeterminedthreshold value may be excluded as deviant values.

That is, a local feature amount U(t+1) in the spherical image B at timet+1 can be expressed in the following expression 201 using a localfeature amount U(t) in the spherical image A at the immediately previoustime t and a change amount F(t+1) of the local feature amount betweenthe spherical images A and B.[Math. 1]U(t+1)=U(t)+F(t+1)  (expression 201)

Next, the image correction unit 1130 converts the local feature amounts(U(t), U(t+1)) calculated by mapping the spherical images A and B to acelestial sphere of equidistant cylindrical projection into3-dimensional feature amounts P(t) and P(t+1). Subsequently, the imagecorrection unit 1130 estimates conversion Mat(t+1) from the3-dimensional feature amounts P(t) to P(t+1) through 3-dimensionalaffine estimation.[Math. 2]P(t+1)=P(t)×Mat(t+1)  (expression 202)

Accordingly, the image correction unit 1130 can estimate the conversionMat(t+1) from P(t) to P(t+1). Further, the image correction unit 1130can calculate rotation Q(t+1) from the spherical image A to thespherical image B based on the estimated conversion Mat(t+1). The imagecorrection unit 1130 may determine whether the estimation of therotation succeeds by calculating an estimation error of the calculatedrotation Q(t+1). The image correction unit 1130 may estimate therotation again when the image correction unit 1130 determines that theestimation fails.

Accordingly, the image correction unit 1130 can generate a sphericalimage C in which the rotation is corrected by performing rotationsubjected to reverse conversion of the estimated rotation Q(t+1) on thespherical image B. The image correction unit 1130 may integrate therotation from the spherical image at a predetermined time and correctthe circumferential captured image based on the integrated rotation.

The local feature amounts focused on by the image correction unit 1130are not particularly limited, but known local feature amounts can beused. As the known local feature amounts, for example, Scale InvariantFeature Transform (SIFT) or the like can be exemplified.

The direction control unit 1190 is a processing unit that controls thedisplay of the circumferential captured image based on a movementdirection of the moving object. Specifically, the direction control unit1190 controls a display image displayed in a display region viewed bythe user so that the reference direction of the display image viewed bythe user matches the movement direction of the moving object. Morespecifically, when an angle difference between the moving direction ofthe moving object and the reference direction of the moving object iswithin a threshold value (for example, 15° to either side for a total of30°), the direction control unit 1190 controls a display field angle ofthe image displayed in the display region viewed by the user so that thereference direction of the display image viewed by the user matches thereference direction or the movement direction of the moving object.

Here, the reference direction of the moving object is, for example, afront direction of the moving object (a human being, a self-propelledobject, a flying object, or the like). The reference direction of thedisplay image viewed by the user is, for example, a field angledirection of the display image displayed in the display region in frontof the user when the user faces in the front direction.

That is, the direction control unit 1190 can control the display fieldangle of the display image so that the front direction of the movingobject matches the field angle direction of the display image displayedin the display region in front of the user when the user faces in thefront direction. Therefore, the direction control unit 1190 can resolveunnaturalness of the display image occurring when the moving objectchanges its movement direction.

For example, the direction control unit 1190 may calculate the movementdirection of the moving object from positional information of the movingobject acquired through positioning using the Global Positioning System(GPS) or Wi-Fi (registered trademark). For example, the directioncontrol unit 1190 may calculate the movement direction of the movingobject from information detected by various sensors such as ageomagnetic sensor and an acceleration sensor. The direction controlunit 1190 may calculate the movement direction of the moving object fromthe circumferential captured images captured by the imaging device.

Here, in the embodiment, the direction control unit 1190 preferablycalculates the movement direction of the moving object from thecircumferential captured images that were captured. In theconfiguration, the direction control unit 1190 can calculate themovement direction of the moving object more efficiently when therotation in the image processing by the image correction unit 1130 iscalculated.

Specifically, when the moving object (more specifically, the imagingdevice) rotates while translating, a translation component is consideredto be superimposed on the change amount F(t+1) of the local featureamount U(t) of the spherical image A and the local feature amount U(t+1)of the spherical image B described in the image correction unit 1130 inaddition to the rotation component. Accordingly, as described above,after the rotation Q(t+1) is estimated by the image correction unit1130, the direction control unit 1190 calculates a difference betweenthe 3-dimensional feature amount P(t+1) of the spherical image B and the3-dimensional feature amount P(t)*Q(t+1) obtained through the estimatedrotation Q(t+1) of the spherical image A. Accordingly, a translationcomponent T(t+1) between the spherical image B and the spherical image Acan be estimated as a difference between P(t+1) and P(t)*Q(t+1)(expression 203).[Math. 3]T(t+1)P(t+1)−P(t)*Q(t+1)  (expression 203)

The translation of the circumferential captured image can be consideredto be generated by movement of the moving object which is being imaged.Accordingly, the movement direction of the moving object can beestimated as an opposite direction to the translation direction of thecircumferential captured image, and thus the direction control unit 1190can estimate the opposite direction of the estimated movement directionof the translation component T(t+1) as the movement direction of themoving object.

Further, the direction control unit 1190 preferably performs a timeaveraging process on the calculated T(t+1) and estimates the movementdirection of the moving object from the translation component Tsubjected to the time averaging. This is because when the movementdirection of the moving object is frequently changed, there is apossibility of the user feeling “nausea” when the direction control unit1190 frequently rotates the display field angle of the display imageviewed by the user. It is needless to say that the direction controlunit 1190 preferably uses the movement direction of the moving objectaveraged at a predetermined time as the movement direction of the movingobject even when the movement direction of the moving object iscalculated with another method.

In accordance with the foregoing method, the direction control unit 1190calculating the movement direction of the moving object changes thedisplay field angle of the display image displayed in the display regionviewed by the user so that the movement direction of the moving objectmatches the reference direction of the display image viewed by the user.In particular, when an angle difference between the movement directionof the moving object and the front direction of the moving object iswithin a threshold value (for example, 15° to either side for a total of30°), the direction control unit 1190 preferably controls the displayimage so that the captured image in the front direction of the movingobject is displayed in front of the user when the user faces in thefront direction. Accordingly, since the direction control unit 1190 canmatch the front direction of the moving object with the field angledirection of the display image displayed in front when the user faces inthe front direction, it is possible to resolve the unnaturalness of thedisplay image occurring when the moving object changes its movementdirection.

Hereinafter, a method of rotating the field angle of the image displayedin the display region viewed by the user to match the movement directionof the moving object in order for the direction control unit 1190 tomatch the movement direction of the moving object with the referencedirection of the display image viewed by the user will be described morespecifically with reference to FIG. 20.

For example, as illustrated in FIG. 20, a movement direction 211 of amoving object 210 is indicated by an arrow and a field angle directionof a display image 231 viewed by the user is assumed to be indicated byangle display. Further, “0°” in the display image 231 viewed by the useris assumed to be the field angle direction of the display image (thatis, the reference direction of the display image) displayed in front ofthe user when the user faces in the front direction.

Here, as illustrated in FIG. 20, when the moving object 210 changes themovement direction 211 to the right, the direction control unit 1190rotates the display field angle of the circumferential captured image tochange the display image 231 viewed in front by the user so that themovement direction 211 of the moving object and “0°” of the displayimage 231 viewed by the user match. Accordingly, it is possible to matchthe movement direction of the moving object with the field angledirection of the display image displayed in front of the user when theuser faces in the front direction. Therefore, even when the movingobject changes its movement direction, the direction control unit 1190can prevent a first-person viewpoint image viewed by the user fromalways being oriented in one direction and thus an unnatural image frombeing generated.

A speed at which the direction control unit 1190 rotates the displayfield angle of the circumferential captured image to match the referencedirection of the display image viewed by the user with the movementdirection of the moving object is preferably a predetermined speed. Thespeed at which the direction control unit 1190 rotates the display fieldangle of the circumferential captured image will be described withreference to FIG. 21.

FIG. 21 is a graph illustrating temporal changes of the movementdirection of the moving object 210 and the reference direction of thedisplay image 231 viewed by the user. The changes of both of thedirections are indicated as angle changes. As illustrated in FIG. 21,for example, when the movement direction of the moving object 210 ischanged at time P by an operation of turning to the right, the directioncontrol unit 1190 rotates the field angle of the display image 231 atthe predetermined speed so that the reference direction of the displayimage 231 viewed by the user matches the movement direction of themoving object 210. Here, in order for the user not to feel “nausea” orthe unnaturalness due to abrupt rotation of the field angle of thedisplay image 231, the direction control unit 1190 preferably gentlyrotates the field angle of the display image 231 at the predeterminedspeed.

The predetermined speed at which the field angle of the display image231 is rotated may be controlled through selection of the user or may becontrolled at a movement speed or a rotation speed of the moving object210. For example, when the movement speed or the rotation speed of themoving object 210 is fast, the direction control unit 1190 may rotatethe field angle of the display image 231 faster. A correspondencerelation between the movement speed or the rotation speed of the movingobject 210 and the rotation speed of the field angle direction of thedisplay image 231 is preferably stored in advance in the storage unit1180 in the form of a correspondence table, a function, or the like.

The above-described function of controlling the field angle of the imagedisplayed in the display region viewed by the user may be controlled sothat the function is not performed by an input from the user or a userstate.

For example, when the user intentionally desires to view in a differentdirection from the movement direction of the moving object, the functionof the direction control unit 1190 may be controlled so that thefunction is not performed according to an input from the user. The inputfrom the user is, for example, an input operation in which the usermaintains the user manipulation device which is a wearable terminal tofix the user manipulation device with both hands.

When the user views in various directions and the visual line directionis not constantly decided, the function of the direction control unit1190 may be controlled so that the function is not performed.Specifically, when a detected variation amount of the visual linedirection of the user exceeds a threshold value, the function of thedirection control unit 1190 may be controlled so that the function isnot performed. When the variation amount of the visual line direction ofthe user is equal to or less than the threshold value for a given time,the function of the direction control unit 1190 may be controlled sothat the function is performed.

In the embodiment, in the display image supplied to the user, the visualline direction of the moving object (for example, a human being) may bedisplayed or a trajectory of the visual line direction of the movingobject (for example, a human being) may be displayed. When the displayimage (for example, the image viewed by the user) is supplied from theinformation processing device 1100 to the moving object (for example, ahuman being), the visual line direction of the user may be displayed inthe display image or the trajectory of the visual line direction of theuser may be displayed in the display image.

The example of the function of the information processing device 1100according to the embodiment has been described. The foregoingconstituent elements may be configured using general members or circuitsor may be configured by hardware specialized for the functions of theconstituent elements. All of the functions of the constituent elementsmay be performed by a CPU or the like. The hardware configurations ofthe embodiment can be changed appropriately according to technologylevels whenever the embodiment is realized.

A computer program for realizing each function of the informationprocessing device according to the above-described embodiment can becreated to be mounted on a personal computer or the like. Acomputer-readable recording medium in which such a computer program isstored can also be supplied. Examples of the recording medium include amagnetic disk, an optical disc, a magneto-optical disc, and a flashmemory. The computer program may be delivered via, for example, anetwork without using a recording medium.

The image generation unit 1110, the image selection unit 1120, the imagecorrection unit 1130, the direction control unit 1190, the dataacquisition unit 1160, the data supply unit 1170, and the storage unit1180 illustrated in FIG. 18 may be mounted on another device such as acomputer capable of communicating with the information processing device10100 and the information processing device 1100 may cooperate with theother device to realize the foregoing functions.

<Flow of Information Processing Method>

Next, the flow of the information processing method performed by theinformation processing device 1100 according to the embodiment will bedescribed in brief with reference to FIG. 22.

In the information processing method according to the embodiment, thecaptured image data is first acquired from the imaging device (camera)mounted on the moving object (step S171).

Next, the image generation unit 1110 of the information processingdevice 1100 generates the circumferential captured images such as thespherical images or the rectangular images obtained by converting thespherical images into the rectangular form based on the acquiredcaptured image data (step S173).

The image correction unit 1130 of the information processing device 1100performs the image processing on the circumferential captured image toextract the rotation component from the generated circumferentialcaptured image and correct the rotation of the moving object (morespecifically, the imaging device) (step S175).

Subsequently, the direction control unit 1190 of the informationprocessing device 1100 acquires movement direction data of the movingobject (step S177).

Here, the direction control unit 1190 of the information processingdevice 1100 determines whether the angle difference between the movementdirection of the moving object and the reference direction of the movingobject (for example, the front direction of the moving object) is withinthe threshold value (step S179). Here, the threshold value may be, forexample, 30° (that is 15° to either side).

When the angle difference between the movement direction of the movingobject and the reference direction of the moving object exceeds thethreshold value (No in step S179), the image selection unit 1120 of theinformation processing device 1100 selects the image to be viewed by theuser among the circumferential captured images. The display control unit1150 of the information processing device 1100 controls the display ofthe selected image on the display screen of the user manipulation device(step S181).

Conversely, when the angle difference between the movement direction ofthe moving object and the reference direction of the moving object iswithin the threshold value (Yes in step S179), the image selection unit1120 and the display control unit 1150 of the information processingdevice 1100 control the display of the image on the display screen as instep S181 (step S183). Further, the direction control unit 1190 of theinformation processing device 1100 performs display control such thatthe circumferential captured image displayed on the display screen isrotated at the predetermined speed at which the movement direction orthe reference direction of the moving object matches the referencedirection of the user (step S185).

Accordingly, the information processing device 1100 can correct theshake of the circumferential captured image caused due to the rotationof the imaging device and can supply the user with the circumferentialcaptured image which is not unnatural even when the moving objectchanges its movement direction.

The flow of the information processing method according to theembodiment has been described in brief with reference to FIG. 22.

<Modification Example of Information Processing Method>

Next, modification examples of the embodiment will be described withreference to FIG. 23. The modification examples of the embodiment are aninformation processing device and an information processing method ofcontrolling an image correction unit so that rotation correction is notperformed on a circumferential captured image when a motion (movement)of a moving object is synchronized with a motion (movement) of a user.

That is, in the modification examples of the embodiment, the imagecorrection unit 1130 determines whether the motion of the moving objectis synchronized with the motion of the user. When the motion of themoving object is determined to be synchronized with the motion of theuser, the functions of the image correction unit 1130 and the directioncontrol unit 1190 are controlled so that the functions are notperformed.

Specifically, the present inventors have found that “nausea” can bereduced and the sense of presence can be improved when the movement orawareness of a user observing a circumferential captured image issynchronized with a movement experienced by a moving object (forexample, a human being). Accordingly, in the modification examples ofthe embodiment, by not performing rotation correction on thecircumferential captured image when the movement of the moving object(for example, a human being) is synchronized with the movement of theuser, it is possible to supply the image with a greater sense ofpresence to the user. In the modification examples of the invention,when a movement not synchronized between the moving object and the useris detected, “nausea” is prevented from occurring for the user byperforming the rotation correction on the circumferential capturedimage.

A movement for which a motion of the moving object (for example, a humanbeing) is synchronized with a motion of the user is, for example, amovement following a tennis ball.

Here, the “synchronization” of a motion of the moving object and amotion of the user will be described with reference to FIG. 23. FIG. 23is a graph illustrating an angle change of rotation of the head of themoving object and an angle change of rotation of the head of the userover time.

As illustrated in FIG. 23, the “synchronization” of a motion of themoving object and a motion of the user indicates, for example, that thedirections of the rotation of the heads of the moving object and theuser are the same. Specifically, in FIG. 23, a waveform of the rotationof the head of a moving object 210 and a waveform of the rotation of thehead of a user 230 vary in amplitudes and periods, but timings of uppercrests and timings of lower troughs are substantially the same. In thiscase, the movement of the moving object and the movement of the user canbe said to be “synchronized.”

More preferably, the “synchronization” of the motion of the movingobject and the motion of the user indicates a case in which thedirections of the rotation of the heads of the moving object and theuser are the same and rotation amounts of the rotation of the heads ofthe moving object and the user are equal to or greater than apredetermined amount. When the rotation amounts of the rotation of theheads of the moving object and the user are small, there is apossibility of the directions of the rotation of the heads of the movingobject and the user matching unintentionally due to an involuntarymovement or the like. Accordingly, when the rotation amounts of therotation of the heads of the moving object and the user are equal to orgreater than the predetermined amount and the directions of the rotationof the heads of the moving object and the user are the same, the usercan more reliably be said to be consciously “synchronizing” the movementof the moving object with his or her movement. The predetermined amountis, for example, 40° (that is 200 on one side).

In addition to the detection of the rotation of the head of the movingobject and the rotation of the head of the user, it is also possible todetect “synchronization” of a motion of the moving object and a motionof the user.

For example, when a movement direction of a centroid position of amoving object (for example, a human being) matches a movement directionof a centroid position of a user, the motion of the moving object may beconsidered to be “synchronized” with the motion of the user. When aninclination direction of the body of a moving object (for example, ahuman being) matches an inclination direction of the body of a user, themotion of the moving object may be considered to be “synchronized” withthe motion of the user. It is possible to detect the centroid positionsand the inclinations of the bodies of the moving object and the user,for example, using known sensors such as an acceleration sensor and amotion sensor.

For example, when a visual line direction of a moving object matches avisual line direction of a user, a motion of the moving object (forexample, a human being) may be considered to be “synchronized” with amotion of the user. Further, when a gazing point of a moving objectmatches a gazing point of a user, a motion of the moving object may beconsidered to be “synchronized” with a motion of the user. This isbecause awareness or recognition is considered to be substantiallymatched (synchronized) between the moving object (for example, a humanbeing) and the user. It is possible to detect the visual line directionor the gazing point, for example, by a visual line detection function ofa wearable terminal mounted on the moving object (for example, a humanbeing) and the user. The visual line detection function can be realizedaccording to the method described in the first embodiment or a knownmethod.

By combining the above-described methods of detecting the“synchronization” of the motion of the moving object and the motion ofthe user, it is possible to detect the “synchronization” of the motionof the moving object and the motion of the user more reliably.

In the modification examples of the embodiment, when the movement of themoving object (for example, a human being) is synchronized with themovement of the user, the image correction unit 1130 does not performthe rotation correction on the circumferential captured image. Whenmovements not synchronized between the moving object and the user aredetected, the image correction unit 1130 performs the rotationcorrection on the circumferential captured image. The flow of theinformation processing method performed in the modification examples ofthe embodiment will be described with reference to FIG. 24.

Here, in the following description, movements experienced by a movingobject and a user are assumed to be movements between the moving objectand the user or movements in which conscious synchronization ispossible.

In the information processing method according to a modification exampleof the embodiment, the captured image data is first acquired from theimaging device (camera) mounted on the moving object (step S191).

Next, the image generation unit 1110 of the information processingdevice 1100 generates the circumferential captured image, for example,the spherical image or the rectangular image obtained by converting thespherical image into a rectangular form based on the acquired capturedimage data (step S193).

Here, the image correction unit 1130 of the information processingdevice 1100 determines whether the rotation of the moving object (morespecifically, the imaging device) is synchronized with the rotation of aterminal mounted on the user. More specifically, the image correctionunit 1130 of the information processing device 1100 determines whetherthe direction of the rotation of the imaging device is the same as thedirection of the rotation of the terminal mounted on the user (stepS195).

Here, when the rotation of the imaging device is not synchronized withthe rotation of the terminal mounted on the user (No in step S195), theimage correction unit 1130 of the information processing device 1100performs the image processing on the circumferential captured image toextract the rotation component from the circumferential captured imageand correct the rotation of the imaging device (step S197). Here, thedirection control unit 1190 may further control the display image basedon the movement direction of the moving object.

Conversely, when the rotation of the moving object is synchronized withthe rotation of the terminal mounted on the user (Yes in step S195), theimage correction unit 1130 of the information processing device 1100does not perform the image processing on the circumferential capturedimage to correct the rotation of the imaging device.

Thereafter, the image selection unit 1120 of the information processingdevice 1100 selects the image to be viewed by the user among thecircumferential captured images. The display control unit 1150 of theinformation processing device 1100 controls the display of the selectedimage on the display screen of the user manipulation device (step S199).

Accordingly, the information processing device 1100 can supply thecircumferential captured image with a higher sense of presence to theuser when the user observes the circumferential captured image whileexperiencing the motion synchronized with the moving object.

The flow of the information processing method according to themodification example of the embodiment has been described in brief abovewith reference to FIG. 24.

<Conclusion>

In this way, in the information processing device and the informationprocessing method according to the embodiment, a video surrounding themoving object can be observed as the circumferential captured images inreal time, and thus the user can obtain the sense of presence just as ifthe user were in the same location as the moving object. By performingthe foregoing correction process, the shake of the images caused due tothe rotation movement of the moving object is suppressed and the changein the movement direction of the moving object can be reflected in thedisplay image. Therefore, occurrence of motion sickness (nausea) issuppressed and the user can view the more natural first-person capturedimage.

Further, in the information processing device and the informationprocessing method according to the modification examples of theembodiment, the circumferential captured image with the higher sense ofpresence can be supplied to the user when the user observes thecircumferential captured image while experiencing the motionsynchronized with the moving object.

The information processing device and the information processing methodaccording to the second embodiment of the present disclosure have beendescribed in detail above.

Third Embodiment

In the related art, many technologies for acquiring real scenery with aplurality of cameras, recombining the scenery in a virtual space, andallowing a user to view the space have been developed. Examples of thetechnologies include a technology for enabling an image captured by acamera mounted on a traveling automobile to be moved inside an image inassociation with positional information and a technology for mounting acamera capable of capturing an omnidirectional image and enabling avisual line direction to be changed freely from a certain location. Bythese technologies, a circumferential image can be viewed while movingin a virtual space. On the other hand, although an observation point ismoved to a nearby observation point and an omnidirectional image can beobserved in these technologies since photographing spots are discrete, aviewpoint may not be moved smoothly. Accordingly, images realized bythese technologies are different from images reproduced as if a userwere at the location.

There is a method called a ray-space theory (Light Field) for enabling aviewpoint position to be moved by imaging a space with cameras disposedin a lattice form. In the ray-space theory, after a space isphotographed by a camera array disposed in the lattice form, pixelsincluded in a captured image are spatially projected to a space havingtwo projection planes, as illustrated in FIG. 25.

Accordingly, light (for example, ray 1 in FIG. 25) passing through arectangular space formed by projection planes A and B can be expressedas one point in a 4-dimensional space that is realized with L(u, v, s,t) using coordinates (u, v) indicating a position on the projection Aand coordinates (s, v) indicating a position on the projection plane B.By expressing the light passing through the two projection planes in theforegoing 4-dimensional space, it is possible to recombine an image froma certain viewpoint from the 4-dimensional space illustrated in FIG. 25.

For example, in FIG. 25, a point a indicates a viewpoint and a point bindicates an observation point viewed from the viewpoint a. The light(ray 2) flowing from the viewpoint a to the observation point b can beexpressed by the foregoing L. Accordingly, on any projection planelocated between the projection planes A and B, a visual field from theviewpoint a can be reproduced by repeating the same process at thepoints which form the projection plane.

However, in the configuration of the ray space illustrated in FIG. 25,there is still a restriction on the degree of freedom of the position ofa viewpoint. For example, since a visual line (ray 3) from a viewpoint cin FIG. 25 does not penetrate through the projection planes A and B, thevisual line may not be expressed as a point in the ray space L.Accordingly, in the recombination of the visual field based on the rayspace illustrated in FIG. 25, there is a restriction on a position.

Accordingly, the present inventors, through diligent investigation, havecome up with a technology capable of recombining a visual field withouta restriction and thus finalized an information processing device to bedescribed below.

<Example of Configuration of System>

FIG. 26 is a diagram illustrating a schematic configuration of a systemaccording to a third embodiment of the present disclosure. Asillustrated in FIG. 26, a system 20 according to the embodiment includesan imaging device 800 and an information processing device 2000.

The restriction in the ray-space theory illustrated in FIG. 25 is causedbecause the arrayed cameras which are imaging devices are disposed inthe 2-dimensional lattice form. Accordingly, the present inventorsthought to remove the restriction on viewpoints by disposing cameras ina 3-dimensional lattice form. In this case, however, problems that (1)certain cameras conceal the visual fields of the other cameras and (2)the number of necessary cameras is too large may occur.

Accordingly, the present inventors investigated self-propelled typeimaging devices used for imaging and consequently came up with the ideathat a space can be imaged in a 3-dimensional lattice form using animaging device 800 which can freely move in a space while avoiding theforegoing problems. The imaging device 800 will be described in detailagain below.

In the system 20 according to the embodiment, a plurality of pieces ofcaptured data captured by the self-propelled imaging device 800 aresupplied to the information processing device 2000 and an imagereconstruction process is performed based on the plurality of pieces ofcaptured data in the information processing device 2000.

The information processing device 2000 performs the image reconstructionprocess based on the plurality of pieces of captured data captured bythe imaging device 800. When the information processing device 2000performs the image reconstruction process, the information processingdevice 2000 generates a 3-dimensional imaged space using the pluralityof pieces of captured data captured by the imaging device 800. Theinformation processing device 2000 performs the image reconstructionprocess using the generated imaged space or a ray space furthergenerated based on the generated imaged space. For example, theinformation processing device 2000 may be configured by at least one ofthe server 100 and the client devices 200 to 700 included in the system10 according to the first embodiment described with reference to FIG. 1.In this case, the server 100 and the client devices 200 to 700 areindependently or cooperatively realized as the information processingdevice 2000 to be described in detail below in terms of the entiresystem 20. It is needless to say that the information processing device2000 may be configured by any of various independent computers. Sincethe hardware device configuration of the information processing device2000 is the same as that in FIG. 2, the detailed description thereofwill be omitted below. The configuration of the information processingdevice 2000 will be described in detail again below.

<Configuration of Imaging Device>

The imaging device 800 according to the embodiment may be configured asa single imaging device such as a robot which can freely move in aspace. However, the self-propelled type imaging device 800 with arrayedcameras illustrated in FIG. 27A or 27B is preferably used.

In the imaging device 800 illustrated in FIG. 27A, cameras 801 arearranged to be disposed in a 1-dimensional direction. At least oneposition recognition marker 803 for specifying an imaging position isinstalled above the cameras 801. The cameras 801 and the positionrecognition markers 803 are supported by a portable stand 805 havingvehicle wheels. The cameras 801 or the stand 805 are controlled by acontrol computer 807 which is installed on the stand 805 and includes abattery. In the imaging device 800 illustrated in FIG. 27B, the cameras801 illustrated in FIG. 27A are disposed in an array form in2-dimensional directions.

In the system 20 according to the embodiment, the imaging device 800illustrated in FIG. 27A or 27B repeats an imaging process while movingin a space.

The cameras 801 installed in the imaging devices 800 illustrated inFIGS. 27A and 27B may be wide angle cameras in which only a field angleof a predetermined direction is set as an imaging visual field, asillustrated in FIG. 28A, or may be omnidirectional cameras in which theentire circumference is an imaging visual field, as illustrated in FIG.28B. However, as will be described below, it is preferable to performimaging as densely as possible in as many directions as possible in thespace. Therefore, an omnidirectional camera illustrated in FIG. 28B ispreferable as the camera 801 installed in the imaging device 800.

The imaging device 800 according to the embodiment repeats imaging whilemoving in the space at a predetermined interval (preferably a constantinterval), as schematically illustrated in FIG. 29. At this time,imaging positions in the space are recorded simultaneously with theimaging by the position recognition markers 803 installed in the imagingdevice 800.

The plurality of pieces of captured data captured by the imaging device800 are output to the information processing device 2000.

<Configuration of Information Processing Device>

The information processing device 2000 according to the embodimentincludes an imaged space generation unit 201, a ray space generationunit 2020, and an image reconstruction unit 2030, as illustrated in FIG.30. The information processing device 2000 may further include one of adisplay control unit 2040, a data acquisition unit 2050, a data supplyunit 2060, and a storage unit 2070. Here, the processing unitsillustrated in FIG. 30 may be realized by any one of the server 100 andthe client devices 200 to 700 or may be distributed to the plurality ofdevices to be realized.

The imaged space generation unit 2010 generates an imaged space in whichinformation indicating positions at which the captured images aregenerated in a space are associated with the corresponding capturedimages, using the plurality of captured images captured by the imagingdevice 800 moving in the space or an imaging device mounted on a movingobject moving in the space. Since the position recognition markers 803are installed along with the cameras 801 in the imaging device 800according to the embodiment, the captured images are easily associatedwith the imaging positions. For example, as schematically illustrated inFIG. 31, the generated imaged space is a 3-dimensional space in whichlattice points correspond to the imaging positions of the imaging device800 or the like and the captured images are associated with the latticepoints.

By generating such an imaged space, it is possible to supply thecaptured image associated with an imaging position (x′, y′, z′) closestto a viewpoint (x, y, z), for example, when there is a designation froma user desiring to view an image viewed from the viewpoint (x, y, z). Inparticular, when the captured image associated with each lattice pointis an omnidirectional image, a designation in the visual line directionis further received from the user and the designated image in the visualline direction is cut from the corresponding omnidirectional image sothat the image can be supplied to be viewed by the user.

To supply the image to the user according to such a method, the numberof lattice points in the imaged space illustrated in FIG. 31 ispreferably as large as possible and the intervals of the mutuallyadjacent lattice points are preferably as short as possible.

In the information processing device 2000 according to the embodiment,not only can a free viewpoint image based on the imaged spaceillustrated in FIG. 31 be supplied, but a free viewpoint image based ona ray space can also be supplied. The ray space is generated by the rayspace generation unit 2020 based on the imaged space generated by theimaged space generation unit 2010.

The ray space generation unit 2020 generates the ray spacecircumscribing the imaged space based on the imaged space generated bythe imaged space generation unit 2010. For example, the ray space may bea rectangular parallelepiped space circumscribing the imaged space, asillustrated in FIG. 32A, or may be a spherical space circumscribing theimaged space, as illustrated in FIG. 32B.

By generating the ray space circumscribing the imaged space illustratedin FIG. 32A or 32B, light penetrating through the inside of the rayspace penetrates any two portions of the ray space. By generating such aray space, all of the rays observed from the inside of the ray space canbe described without limiting observation positions or observationdirections.

For example, as illustrated in FIG. 32A, when the ray space is therectangular parallelepiped space, the light penetrating the ray spacepenetrates two portions among six surfaces of the rectangularparallelepiped. Therefore, a ray L can be defined using the coordinatesof the penetration points of the light on the rectangularparallelepiped. The ray L defined in this way corresponds to one pointin the 4-dimensional space referred to as (u, v, s, t). Here, thecoordinate system differs for each surface of the rectangularparallelepiped. However, as illustrated in the lower portion of FIG.32A, points can all be expressed in the same coordinate systemregardless of what surfaces are penetrated by considering thedevelopment view of the ray space.

For example, as illustrated in FIG. 32B, when the ray space is set as aspherical space, one point on the spherical surface can be convertedinto one point on a rectangular plane, for example, by a known methodsuch as equidistant cylindrical projection. Accordingly, as illustratedin FIG. 32B, even when the ray space is set as the spherical space, theray L can correspond to one point in the 4-dimensional space, as in thecase of FIG. 25A.

The image shown when the space is viewed can be reconstructed inaccordance with the ray space generated in this way. The reconstructionprocess is performed by the image reconstruction unit 2030.

The image reconstruction unit 2030 reconstructs an image correspondingto reconstruction information regarding the visual line direction andthe viewpoint designated by the user based on the ray space generated bythe ray space generation unit 2020 and the reconstruction information.Hereinafter, an image reconstruction method will be described withreference to FIG. 33.

One point of the image viewed by the user can be expressed by a vectorindicated by a viewpoint position and a viewpoint direction from theviewpoint position, as illustrated in FIG. 33. When a vector illustratedin FIG. 33 is specified based on the reconstruction information, theimage reconstruction unit 2030 extends the vector and calculates thepositions of intersection points between the vector and the ray space.Here, when the coordinates of the intersection points are (u, v) and (s,t), the ray L corresponding to pixels corresponding to thereconstruction information (pixels in a reconstructed image) can bereproduced from L(u, v, s, t), as illustrated in FIG. 33.

The image reconstruction unit 2030 can reproduce the image at anyviewpoint designated by the user by repeatedly performing theabove-described process on all of the pixels included in thereconstructed image.

The reconstruction method can be realized with higher precision bycausing the imaging device 800 to densely capture a multi-viewpointimage and generating the ray space. Since a relation between thegenerated ray space and the imaging is simple and the above-describedspace reproduction method is simple, the foregoing process can beperformed at low calculation cost. The reconstruction method accordingto the embodiment can be said to be a general method capable ofconsiderably simply reconstructing an image at any viewpoint and in anydirection since recognition of correspondence points between two imagessuch as multi-viewpoint stereo images in the related art is notnecessary.

The display control unit 2040 controls display content of a displaydevice such as a display installed in the information processing device2000 or outside of the information processing device 2000. Specifically,the display control unit 2040 can allow the user to view the imagedspace generated by the imaged space generation unit 2010, the ray spacegenerated by the ray space generation unit 2020, the image reconstructedby the image reconstruction unit 2030, or the like. Accordingly, theuser can comprehend the image at any viewpoint or in any direction atall times.

The data acquisition unit 2050 acquires captured-image data output fromthe imaging device 800 or data regarding the imaging position oracquires data regarding a user manipulation output from the usermanipulation device, various input mechanisms, or the like. The variouskinds of data acquired by the data acquisition unit 2050 can beappropriately used by the processing units included in the informationprocessing device 2000.

The data supply unit 2060 supplies the various kinds of data (forexample, reconstructed-image data) generated by the informationprocessing device 2000 to a device installed outside of the informationprocessing device 2000. Accordingly, even the device installed outsideof the information processing device 2000 can use various kinds ofinformation generated by the information processing device 2000.

The storage unit 2070 may appropriately record various databases usedfor processes of the imaged space generation unit 2010, the ray spacegeneration unit 2020, the image reconstruction unit 2030, the displaycontrol unit 2040, the data acquisition unit 2050, and the data supplyunit 2060, various programs including applications used for variouscalculation processes performed by these processing units, and variousparameters necessarily stored when certain processes are performed orcourses of interim processes.

The storage unit 2070 can be freely accessed by each processing unitsuch as the imaged space generation unit 2010, the ray space generationunit 2020, the image reconstruction unit 2030, the display control unit2040, the data acquisition unit 2050, and the data supply unit 2060, sothat data can be written or read.

The example of the function of the information processing device 2000according to the embodiment has been described. The foregoingconstituent elements may be configured using general members or circuitsor may be configured by hardware specialized for the functions of theconstituent elements. All of the functions of the constituent elementsmay be performed by a CPU or the like. Accordingly, the configurationsto be used can be changed appropriately according to technology levelswhenever the embodiment is realized.

A computer program for realizing each function of the informationprocessing device according to the above-described embodiment can becreated to be mounted on a personal computer or the like. Acomputer-readable recording medium in which such a computer program isstored can also be supplied. Examples of the recording medium include amagnetic disk, an optical disc, a magneto-optical disc, and a flashmemory. The computer program may be delivered via, for example, anetwork without using a recording medium.

<Flow of Information Processing Method>

Next, the flow of the information processing method performed by theinformation processing device 2000 according to the embodiment will bedescribed in brief with reference to FIGS. 34 and 35.

[Ray Space Generation Process]

First, the flow up to a ray space generation process will be describedwith reference to FIG. 34.

The imaged space generation unit 2010 of the information processingdevice 2000 according to the embodiment acquires the captured datacaptured by the imaging device 800 and generates the imaged spaceillustrated in FIG. 33 based on the captured data and the informationregarding the imaging position associated with the captured data (stepS201).

Next, the ray space generation unit 2020 of the information processingdevice 2000 first performs a definition loop of the ray L indicatedafter step S203 using the imaged space generated by the imaged spacegeneration unit 2010. The definition loop of the ray L is performeduntil the process on all of the pixels (Px, Py) of all of the imagingpositions (Cx, Cy, Cz) is completed.

In the definition loop of the ray L, the ray space generation unit 2020first sets intensity I(Cx, Cy, Cz, Px, Py) in the luminance of eachpixel (Px, Py) of the imaging position (Cx, Cy, Cz) (step S205).Subsequently, the ray space generation unit 2020 calculates coordinates(u, v, s, t) of the ray space based on parameters Cx, Cy, Cz, Px, and Py(step S207). Thereafter, the ray space generation unit 2020 sets L(u, v,s, t)=I(Cx, Cy, Cz, Px, Py) (step S209).

When the above-described definition loop of the ray L ends, the rayspace generation unit 2020 performs a complementary loop of the ray Lindicated after step S211. The complementary loop of the ray L isperformed until the process ends for all of the coordinates (u, v, s, t)of the ray space.

In the complementary loop of the ray L, the ray space generation unit2020 first determines whether L(u, v, s, t) is defined for thecoordinates (u, v, s, t) of the ray space of interest (step S213). WhenL(u, v, s, t) is defined in advance, the ray space generation unit 2020continues the process of step S213 on other coordinates (u, v, s, t) ofthe ray space. Conversely, when L(u, v, s, t) is not defined, the rayspace generation unit 2020 performs a known complementary process basedon the defined L near (u, v, s, t) and calculates L(u, v, s, t) (stepS215). Accordingly, L(u, v, s, t) at a ray space position at which L isnot defined is complemented based on the defined L(u, v, s, t).

When the above-described complementary loop of the ray L ends, the rayspace illustrated in FIG. 32A or 32B is generated.

[Image Reconstruction Process]

Next, the flow of the image reconstruction process will be describedwith reference to FIG. 35.

The image reconstruction unit 2030 of the information processing device2000 according to the embodiment first specifies a viewpoint position(Cx, Cy, Cz) and the position (Px, Py) of each pixel included in theimage to be reconstructed with reference to information regarding a usermanipulation or the like (step S301).

Next, the image reconstruction unit 2030 performs a luminancecalculation loop indicated after step S303. The luminance calculationloop is performed until the process ends for all of the pixels (Px, Py)included in the image to be reconstructed.

In the luminance calculation loop, the image reconstruction unit 2030first calculates the coordinates (u, v, s, t) of the ray space based onthe parameters (Cx, Cy, Cz, Px, Py) (step S305). Thereafter, the imagereconstruction unit 2030 calculates a pixel value I(Px, Py)=L(Cx, Cy,Cz, Px, Py) of a pixel of interest using L(u, v, s, t) (step S307).

When the above-described complementary loop of the ray L ends, the imageat any position and any viewpoint designated by the user manipulation isreconstructed.

The image reconstruction unit 2030 outputs data of the imagereconstructed in this way via the display control unit 2040 or the datasupply unit 2060 (step S309). Accordingly, the reconstructed image isviewed by the user.

The flow of the information processing method according to theembodiment has been described in brief with reference to FIGS. 34 and35.

<Conclusion>

In this way, in the information processing device and the informationprocessing method according to the embodiment, a predetermined ray spaceis generated based on a plurality of pieces of captured-image datacaptured by an imaging device moving in the space and an image at anyviewpoint and in any direction designated by a user is reconstructedbased on the generated ray space. In the ray space generated by theinformation processing device and the information processing methodaccording to the embodiment, all of the rays observed from the inside ofthe ray space can be described without limiting an observation positionor an observation direction. Therefore, it is possible to recombinevisual fields without restrictions.

The preferred embodiment(s) of the present disclosure has/have beendescribed above with reference to the accompanying drawings, whilst thepresent disclosure is not limited to the above examples. A personskilled in the art may find various alterations and modifications withinthe scope of the appended claims, and it should be understood that theywill naturally come under the technical scope of the present disclosure.

In addition, the effects described in the present specification aremerely illustrative and demonstrative, and not limitative. In otherwords, the technology according to the present disclosure can exhibitother effects that are evident to those skilled in the art along with orinstead of the effects based on the present specification.

Additionally, the present technology may also be configured as below.

(1)

An information processing device including:

a control unit configured to perform control in a manner that a displayimage generated based on image information which is generated throughimaging of an imaging device mounted on a moving object moving in aspace, imaging-device posture information which is information regardinga posture of the imaging device, and user view information which isobtained from a user manipulation device manipulated by a user andspecifies a region that the user desires to view is displayed in adisplay region viewed by the user.

(2)

The information processing device according to (1), further including:

an image generation unit configured to generate circumferential capturedimages which are captured in circumference of a position at which themoving object is present using captured images included in the imageinformation; and

an image selection unit configured to select a captured imagecorresponding to the user view information among the circumferentialcaptured images as a user view image based on the user view informationand the circumferential captured images generated by the imagegeneration unit.

(3)

The information processing device according to (2), further including:

an image correction unit configured to perform correction on thecircumferential captured image in a manner that a change in thecircumferential captured image accompanying a change in a visual linedirection of the imaging device is suppressed, based on theimaging-device posture information, when the visual line direction ofthe imaging device is changed.

(4)

The information processing device according to (3),

wherein the image correction unit controls a degree to which thecorrection is performed according to correction application informationindicating an application degree of the correction obtained from theuser manipulation device.

(5)

The information processing device according to (4),

wherein the image correction unit controls the degree to which thecorrection is performed for each of rotation coordinate axes mutuallyindependently defined with respect to the imaging device according tothe correction application information.

(6)

The information processing device according to any one of (3) to (5),

wherein the image correction unit performs correction in a manner thatthe circumferential captured image after a change in a visual linedirection of the moving object is reversely rotated according to amagnitude of a rotation angle accompanying the change in the visual linedirection of the imaging device.

(7)

The information processing device according to any one of (3) to (6),

wherein the image correction unit performs the correction so that localfeature amounts match before and after the change in the visual linedirection of the imaging device.

(8)

The information processing device according to any one of (2) to (7),

wherein the image generation unit generates, as the circumferentialcaptured image, an omnidirectional image at the position at which themoving object is present or a converted image obtained by converting theomnidirectional image into a rectangular image.

(9)

The information processing device according to any one of (1) to (8),

wherein the imaging-device posture information is information regardingrotation of the imaging device.

(10)

The information processing device according to any one of (2) to (9),

wherein the user view information is information specifying a displayfield angle that the user desires to view in the circumferentialcaptured image.

(11)

The information processing device according to (2), further including:

a visual line information generation unit configured to generate visualline information indicating a visual line direction of the imagingdevice based on the imaging-device posture information,

wherein the control unit displays an object indicating the visual linedirection of the imaging device indicated with the visual lineinformation along with the user view image using the visual lineinformation generated by the visual line information generation unit.

(12)

The information processing device according to (10) or (11),

wherein the control unit

generates posture information in which a change in the posture of theimaging device is visualized using rotation information regardingrotation accompanying a change in a visual line direction of the imagingdevice and calculated based on the imaging-device posture information,and

performs control in a manner that the display image is displayed in thedisplay region viewed by the user with the generated posture informationsuperimposed on the display image.

(13)

The information processing device according to (12), further including:

an image correction unit configured to perform correction on thecircumferential captured image in a manner that a change in thecircumferential captured image accompanying the change in the visualline direction of the imaging device is suppressed when the visual linedirection of the imaging device is changed without a change in aposition of the imaging device,

wherein the control unit superimposes an object indicating the postureinformation on the circumferential captured image corrected by the imagecorrection unit.

(14)

The information processing device according to (13),

wherein the control unit superimposes at least one of an object that isrotated with a rotation movement accompanying the change in the visualline direction of the imaging device and an object that is not rotatedon the display image.

(15)

The information processing device according to any one of (12) to (14),

wherein the control unit visualizes the change in the posture of theimaging device using the rotation information by fixing a motion of acoordinate system fixed to the space in which the imaging device ispresent and changing the coordinate system fixed to the imaging deviceor by changing the motion of the coordinate system fixed to the space inwhich the imaging device is present and fixing the motion of thecoordinate system fixed to the imaging device.

(16)

The information processing device according to (15),

wherein the control unit generates the display image corresponding to acase in which the space is virtually viewed from a different positionfrom a center of the coordinate system fixed to the space, whenvisualizing the change in the posture of the imaging device using therotation information.

(17)

The information processing device according to any one of (12) to (16),

wherein the control unit associates an annotation with a correspondencespot of a specific position of the display image in a coordinate systemfixed to the space in which the imaging device is present when theannotation is requested to be added to the specific position from theuser manipulation device.

(18)

The information processing device according to any one of (12) to (17),

wherein, according to the rotation information, the control unitcontrols at least one of a reproduction speed or a display field anglewhen the display image is displayed in the display region viewed by theuser.

(19)

The information processing device according to any one of (12) to (18),

wherein the control unit generates a display image in a case of virtualviewing of the space from an arbitrary position designated from the usermanipulation device, centering on the designated arbitrary position.

(20)

The information processing device according to any one of (13) to (19),

wherein the control unit changes setting for a correction process in theimage correction unit and setting for a superimposition process for theposture information in the control unit based on a manipulationperformed on at least one of the imaging device and the usermanipulation device.

(21)

The information processing device according to (2), further including:

an image correction unit configured to perform correction on thecircumferential captured image in a manner that a change in thecircumferential captured image accompanying rotation of the imagingdevice is suppressed.

(22)

The information processing device according to (21),

wherein the control unit controls display of the display image based ona movement direction of the moving object wearing the imaging device.

(23)

The information processing device according to (22),

wherein when an angle difference between the movement direction of themoving object and a reference direction of the moving object is within athreshold value, the control unit rotates the display field angle of thedisplay image displayed in the display region viewed by the user at apredetermined speed so that a reference direction in the display imagematches the reference direction or the movement direction of the movingobject.

(24)

The information processing device according to (23),

wherein the predetermined speed is controlled based on at least one of amovement speed and a rotation speed of the moving object.

(25)

The information processing device according to any one of (21) to (24),

wherein the user manipulation device is a wearable device which ismounted on the user and in which the display image is displayed, and

wherein when a rotation direction of the moving object and a rotationdirection of the user manipulation device match, the image correctionunit does not perform correction in a manner that the change in thecircumferential captured image is suppressed.

(26)

The information processing device according to (25),

wherein when rotation amounts of the imaging device and the usermanipulation device are equal to or less than a threshold value, theimage correction unit performs correction in a manner that the change inthe circumferential captured image is suppressed.

(27)

The information processing device according to any one of (1) to (26),

wherein the user manipulation device is a wearable device mounted on theuser, and

wherein the user view information is generated according to a visualline direction of the user detected by the wearable device.

(28)

The information processing device according to any one of (1) to (27),

wherein the moving object is one of a human being different from theuser manipulating the user manipulation device, a self-propelled objectthat propels itself in the space, and a flying object that flies in thespace.

(29)

The information processing device according to (1), further including:

an acquisition unit configured to acquire intermediate image informationgenerated based on the image information and the imaging-device postureinformation,

wherein the control unit performs control in a manner that the displayimage generated based on the intermediate image information and the userview information is displayed in the display region viewed by the user.

(30)

An information processing method including:

performing control in a manner that a display image generated based onimage information which is generated through imaging of an imagingdevice mounted on a moving object moving in a space, imaging-deviceposture information which is information regarding a posture of theimaging device, and user view information which is obtained from a usermanipulation device manipulated by a user and specifies a region thatthe user desires to view is displayed in a display region viewed by theuser.

(31)

A program causing a computer to realize a control function of:

performing control in a manner that a display image generated based onimage information which is generated through imaging of an imagingdevice mounted on a moving object moving in a space, imaging-deviceposture information which is information regarding a posture of theimaging device, and user view information which is obtained from a usermanipulation device manipulated by a user and specifies a region thatthe user desires to view is displayed in a display region viewed by theuser.

REFERENCE SIGNS LIST

-   1000, 1100, 2000 information processing device-   1010, 1110 image generation unit-   1020, 1120 image selection unit-   1030, 1130 image correction unit-   1040 moving-object visual line information generation unit-   1050, 1150 display control unit-   1060, 1160, 2010 data acquisition unit-   1070, 1170, 2060 data supply unit-   1080, 1180, 2070 storage unit-   1190 direction control unit-   2020 imaged-space generation unit-   2030 ray space generation unit-   2040 image reconstruction unit

The invention claimed is:
 1. An information processing devicecomprising: circuitry configured to control a display terminal to:acquire a captured image obtained by an imaging device via a networkserver; display a partial image of the captured image in a displayregion of the display terminal, wherein the partial image corresponds toa part of the captured image that is specified based on viewinformation, the view information is obtained from the display terminal,and the captured image is wider than the partial image; performcorrection on the captured image so as to suppress a change in thecaptured image resulting from a change of a posture of the imagingdevice; and display a superimposed image over the partial image when thecorrection on the captured image is performed, wherein the superimposedimage represents a relative coordinate system that is fixed to theimaging device.
 2. The information processing device according to claim1, wherein in the correction performed on the captured image, thecircuitry is further configured to: acquire a rotation component of achange from a first captured image to a second captured image, whereinthe second captured image is obtained by the imaging device after thefirst captured image; and reversely rotate the second captured image,based on the rotation component, to match the first local featureamounts of the first captured image with second local feature amounts ofthe second captured image.
 3. The information processing deviceaccording to claim 1, wherein the superimposed image indicates theposture of the imaging device based on a signal from at least one sensorthat is installed in an apparatus including the imaging device.
 4. Theinformation processing device according to claim 1, wherein thesuperimposed image includes an object that is not moved on the relativecoordinate system and indicates at least one of a numerical value or aletter.
 5. The information processing device according to claim 1,wherein the circuitry is further configured to control the displayterminal to display the superimposed image to rotate, when viewed from auser of the display terminal, in a real space in which the imagingdevice is present.
 6. The information processing device according toclaim 5, wherein the circuitry is further configured to control thedisplay terminal to offset a viewpoint of the user from a position ofthe imaging device in the real space.
 7. The information processingdevice according to claim 1, wherein the circuitry is further configuredto fix an annotation object to a specific position of an absolutecoordinate system, which is fixed to a real space in which the imagingdevice is present, while suppressing the change in the captured imageresulting from the change of the posture of the imaging device.
 8. Theinformation processing device according to claim 1, wherein thecircuitry is further configured to control the display terminal toreduce a reproduction speed of the partial image in accordance with anincrease of a movement amount of the imaging device.
 9. The informationprocessing device according to claim 1, wherein the circuitry is furtherconfigured to control the display terminal to change a viewpoint a userof the display terminal based on a designation from the user along adirection including a present viewpoint of the user.
 10. The informationprocessing device according to claim 1, wherein the circuitry is furtherconfigured to control the display terminal to: set, based on amanipulation performed on at least one of the imaging device and thedisplay terminal, a visualization state of the captured image and thesuperimposed image to one of a first state or a second state, wherein inthe first state, the correction on the captured image is not performedand a position of the superimposed image is prevented from moving on thedisplay region of the display terminal even when the posture of theimaging device is changed, and wherein in the second state, thecorrection on the captured image is performed and the position of thesuperimposed image is moved on the display region of the displayterminal to be rotated in a real space in which the imaging device ispresent when viewed from.
 11. The information processing deviceaccording to claim 1, wherein the is display terminal comprises awearable device configured to be mounted on a user, and wherein thecircuitry is further configured to change the view information accordingto a change of an orientation of the wearable device to provide afirst-person viewpoint image to the user independently from a change ofthe posture of the imaging device.
 12. The information processing deviceaccording to claim 1, wherein the imaging device is mounted onto one ofa self-propelled object that propels itself in a real space in which theimaging device is present or a flying object that flies in the realspace.
 13. The information processing device according to claim 1,wherein the imaging device is configured to be attached onto a part of abody of a person who is different from a user of the display terminal.14. The information processing device according to claim 13, wherein theimaging device includes a plurality of cameras, and wherein the capturedimage includes a plurality of captured images obtained from theplurality of cameras.
 15. The information processing device according toclaim 14, wherein the imaging device comprises a wearable cameraconfigured to communicate with the display terminal via the networkserver in real time.
 16. The information processing device according toclaim 15, wherein the plurality of captured images is a plurality ofcircumferential captured images each of which represents a circumferenceof the wearable camera, and wherein the circuitry is further configuredto select, based on the view information, at least one of the pluralityof circumferential captured images as the partial image withoutperforming collation of feature points between the plurality of thecaptured images.
 17. The information processing device according toclaim 1, wherein the circuitry is further configured to control thedisplay terminal to change, in the correction on the captured image, thepartial image to gradually follow a rotation movement of the imagingdevice.
 18. The information processing device according to claim 1,wherein the circuitry is further configured to: determine whether animaging direction of the imaging device is changed while a position ofthe imaging device in a real space is not substantially changed; andperform the correction on perform the correction on the captured imageso as to suppress the change in the captured image based ondetermination that the imaging direction of the imaging device ischanged while the position of the imaging device in the real space isnot substantially changed.
 19. The information processing deviceaccording to claim 18, wherein the circuitry is further configured todetermine, based on a magnitude of a rotation angle in accordance with achange of the imaging direction, whether the imaging direction of theimaging device is changed while the position of the imaging device. 20.An information processing method comprising: controlling a displayterminal to: acquire a captured image obtained by an imaging device viaa network server; display a partial image of the captured image in adisplay region of the display terminal, wherein the partial imagecorresponds to a part of the captured image that is specified based onview information, the view information is obtained from the displayterminal, and the captured image is wider than the partial image;perform correction on the captured image so as to suppress a change inthe captured image resulting from a change of a posture of the imagingdevice; and display a superimposed image over the partial image when thecorrection on the captured image is performed, wherein the superimposedimage represents a relative coordinate system that is fixed to theimaging device.
 21. A non-transitory computer-readable medium havingembodied thereon a program, which when executed by a computer causes thecomputer to execute a method, the method comprising: controlling adisplay terminal to: acquire a captured image obtained by an imagingdevice via a network server; display a partial image of the capturedimage in a display region of the display terminal, wherein the partialimage corresponds to a part of the captured image that is specifiedbased on view information, the view information is obtained from thedisplay terminal, and the captured image is wider than the partialimage; perform correction on the captured image so as to suppress achange in the captured image resulting from a change of a posture of theimaging device; and display a superimposed image over the partial imagewhen the correction on the captured image is performed, wherein thesuperimposed image represents a relative coordinate system that is fixedto the imaging device.